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Developing  Effective  Strategies  and  Performance  Metrics  for 
Automatic  Target  Recognition 

(Work  Done  at  the  University  of  South  Alabama) 

Abstract 

The  final  report  that  summarizes  the  work  performed  at  University  of  South  Alabama  (USA). 
Four  different  target  tracking  algorithms  and  two  data  fusion  algorithms  have  been  developed  for 
single/multiple  target  detection  and  tracking  purposes.  Each  tracking  algorithm  utilizes  various 
properties  of  targets  and  image  frames  of  a  given  sequence.  The  data  fusion  algorithms  employ 
complementary  features  of  two  or  more  of  the  above  mentioned  algorithms.  Thus,  the  data  fusion 
technique  has  been  found  to  yield  the  best  performance. 

The  performance  of  the  above  mentioned  algorithms  was  evaluated  using  two  approaches  - 
evaluation  based  on  the  input  scene  data  complexity,  and  evaluation  based  on  the  correlation 
output  produced  by  each  algorithm.  We  developed  two  new  performance  metrics  to  evaluate  the 
effect  of  input  plane  data  complexity  on  the  performance  of  the  algorithms.  To  evaluate  the 
output  produced  by  each  of  the  aforementioned  algorithms,  we  used  four  performance  metrics, 
such  as  peak-to-sidelobe  ratio,  peak-to-correlation  energy,  peak-to-clutter  ratio,  and  Fisher  ratio. 

Finally,  we  investigated  the  problem  of  target  detection  in  the  initial  frame  of  a  video  sequence 
using  two  techniques,  namely  feature  vectors  and  multilevel  data  fusion,  assuming  that  no  target 
information  is  known  a  priori. 


1.  Multiple  Target  Tracking  Algorithms 

We  developed  four  new  multiple  target  tracking  algorithms  which  are  based  on  fringe-adjusted 
joint  transform  correlation  (FJTC),  intensity  variation  function  (IVF)  and  template  matching 
(TM)  techniques.  The  performance  of  these  algorithms  were  tested  using  50  real  life  forward 
looking  infrared  (FLIR)  image  sequences  supplied  by  the  Army  Missile  Command  (AMCOM). 


1.1  FJTC,  IVF  and  TM  (FJTC-IVF-TM)  Based  Multiple  Target  Tracking 

In  this  section,  we  introduce  FJTC-IVF-TM  based  multiple  target  tracking  algorithm  which 
includes  frame  preprocessing,  motion  estimation  and  target  tracking.  In  the  initial  stage,  we 
utilized  the  FJTC  technique  [1],  Using  FJTC  results,  the  current  frame  of  the  image  sequence  is 
recovered  after  global  motion  compensation.  For  the  tracking  algorithm,  IVF  and  TM  techniques 
have  been  used.  The  IVF  based  target  tracking  approach  primarily  utilizes  target  intensity 
information  associated  with  the  previous  frame  and  the  current  frame.  Initially,  a  target  reference 
window  and  a  subframe  are  segmented  from  consecutive  frames.  Then  the  IVF  is  generated  by 
sliding  target  window  inside  the  frame  or  the  subframe,  which  yields  maximum  peak  value  for  a 
match  between  the  intensity  variations  of  the  known  reference  target  and  the  unknown  candidate 
target  window  in  the  frame  or  the  subframe  as  shown  in  Fig.  1 . 
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(a)  (b) 


Fig.  1.  Intensity  variation  function  based  target  detection  in  a  sequence  (lwir_1913),  a)  Reference  target  window 
segmentation  from  previous  frame,  b)  Subframe  segmentation  from  the  subsequent  frame,  c)  IVF  result ,  d)  Flighest 

peak  produced  in  the  IVF  output  plane. 

From  Fig.  1,  it  is  evident  that  the  candidate  target  coordinates  can  be  obtained  from  the 
maximum  peak  value.  When  the  candidate  target  coordinates  are  less  reliable,  the  TM  technique 
is  triggered.  It  generates  the  highest  peak  value  based  on  the  similarity  between  reference  target 
and  candidate  target  by  sliding  target  window  inside  the  subframe  [2],  Using  the  new  target 
coordinates,  target  model  is  updated  for  target  detection  in  the  next  frame  as  well  as  the  tracking 
process.  The  aforementioned  steps  are  repeated  until  all  frames  of  the  sequence  are  processed. 
The  target  tracking  results  obtained  using  the  FJTC-IYF-TM  algorithm  is  shown  in  Appendix  A. 

1.2  Normalized  Crosscorrelation,  IVF  and  TM  (CC-IVF-TM)  Based  Multiple  Target 
Tracking 

In  this  algorithm,  we  proposed  a  new  frame  preprocessing  algorithm  using  regional  segmentation 
and  normalized  crosscorrelation  based  template  modeling  technique.  The  regional  segmentation 
approach  alleviates  the  effects  of  several  detrimental  factors  such  as  smoke  motion  and  effects  of 
background  while  reducing  the  computation  time.  In  addition,  the  template  modeling  technique 
enhances  processing  speed  and  increases  the  accuracy  of  the  tracking  algorithm. 

For  estimating  the  shift  distance,  template  modeling  algorithm  is  used  which  yields  the  highest 
value  corresponding  to  the  correlation  coefficients  between  known  reference  image  and 
unknown  input  image.  The  correlation  coefficients  provide  a  measure  of  association  of  the 
relationship  between  two  images.  It  is  defined  as  the  ratio  of  the  observed  covariance  of  two 
standardized  variables,  divided  by  the  highest  possible  covariance.  When  the  observed 
covariance  corresponds  to  the  possible  covariance  value,  the  correlation  operation  generates  a 
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value  of  1,  indicating  a  perfect  match  between  the  two  images.  In  contrast,  a  correlation  value  of 
0  indicates  a  random  relationship  between  the  two  images. 

The  frame  preprocessing  is  achieved  by  determining  the  highest  correlation  coordinate  between 
the  reference  image  and  template  image  obtained  from  the  subframe.  Using  the  frame 
preprocessing,  the  current  frame  of  the  image  sequence  is  recovered.  For  tracking,  we  used  the 
IVF-TM  tracking  algorithm  explained  earlier  in  Section  1.1.  The  multiple  target  tracking  results 
corresponding  to  the  CC-IVF-TM  algorithm  is  depicted  in  Appendix  A. 

1.3.  Normalized  Crosscorrelation,  FJTC,  and  TM  (CC-FJTC-TM)  Based  Multiple  Target 
Tracking 

In  this  tracking  algorithm,  we  used  the  normalized  crosscorrelation  based  preprocessing 
technique  explained  in  Subsection  1.2.  For  tracking  purposes,  we  used  the  FJTC  and  template 
modeling  techniques.  The  FJTC  technique  has  been  found  to  yield  excellent  correlation 
discrimination  between  a  known  reference  and  an  unknown  target  in  an  input  scene.  After 
preprocessing  and  applying  the  FJTC  operation,  a  shift  vector  is  obtained  using  the  difference 
between  the  reference  correlation  peak  and  the  new  correlation  peak  coordinates  as  shown  in  Fig 
2.  The  shift  vector  is  used  to  calculate  new  target  coordinates. 


Correlation  Output 


0  0 


Fig.2  FJTC  based  target  detection  algorithm  scheme. 
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When  the  FJTC  algorithm  yields  poor  results  due  to  blending  or  background  clutter  problems, 
template  modeling  technique  is  employed  as  a  compensation  technique.  Thereafter,  the  target 
model  is  updated  and  is  used  as  a  reference  for  the  subsequent  frame.  This  procedure  is  repeated 
until  all  frames  of  the  sequence  are  processed.  The  target  tracking  results  corresponding  to  the 
CC-FJTC-TM  algorithm  is  included  in  Appendix  A. 

1.4  Modified  SDF  based  Invariant  FJTC  (FJTC-MSDF-FJTC)  Target  Tracking 

This  is  a  vigorous  approach  for  real-time  target  tracking  in  FLIR  imagery  in  the  presence  of 
various  distortions  in  target  features  and  background  properties.  The  proposed  approach  utilizes 
FJTC  technique  for  real-time  estimation  of  target  motion  and  utilizes  a  concept  similar  to  the 
synthetic  discriminant  function  (SDF)  filter  to  update  the  reference  target  model  in  order  to 
alleviate  the  effects  of  noise,  3D  distortions,  and  hazards  generated  by  bad  frames  yielding 
significantly  improved  overall  tracking  performance  [3]. 


Fig.  3.  FJTC  Tracking  Algorithm 

The  FJTC  technique  is  used  for  correlation  due  to  its  low  sensitivity  to  scale  and  rotation 
variations,  small  correlation  side-lobes,  and  low  zero-order  peak.  Moreover,  FJTC  can  be  easily 
implemented  and  is  very  suitable  for  real-time  applications.  Figure  3  shows  the  block  diagram  of 
the  proposed  FJTC  tracking  system. 

A  FLIR  image  generally  smoothes  out  object  edges  and  comers,  and  thus  leads  to  a  reduction  of 
distinct  features.  The  solution  is  to  apply  image  preprocessing  to  both  the  input  and  reference 
images  before  the  application  of  every  FJTC  operation.  Image  preprocessing  involves  various 
procedures  such  as  edge  enhancement  and  local  intensity  normalization.  In  the  proposed  FJTC 
algorithm,  image  segmentation  technique  is  used  to  improve  target  tracking  with  conventional 
FJTC.  In  this  method,  several  steps  of  motion  preprocessing  are  used  to  effectively  handle 
complicated  scenarios  such  as  multiple  independently  moving  objects,  distortions  due  to  skew, 
rotation  as  well  as  other  artifacts. 

In  addition,  a  filter  is  used  to  preserve  the  information  available  from  the  preceding  target 
reference  images  in  order  to  reduce  the  effects  of  noise  and  hazard  associated  with  bad  frames,  as 
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well  as  other  artifacts  as  shown  in  Fig.  4.  The  results  corresponding  to  the  modified  SDF  based 
invariant  FJTC  target  tracking  algorithm  are  illustrated  in  Appendix  A  (Algorithm  #4).  Finally 
each  tracking  algorithm  results  are  summarized  in  Appendix  B. 


Fig.4  Modified  SDF  model  based  reference  target  update  scheme. 

2.  Data/Decision  Fusion 

We  proposed  a  novel  data/decision  fusion  algorithm  for  target  tracking  in  FLIR  image 
sequences.  The  algorithm  allows  the  fusion  of  complementary  preprocessing  and  tracking 
algorithms.  We  identified  three  modes  that  contribute  to  the  failure  of  the  tracking  system:  (1) 
the  motion  failure  mode,  (2)  the  tracking  failure  mode,  and  (3)  the  reference  image  distortion 
induced  failure  mode.  The  strategy  in  the  design  of  the  proposed  algorithm  is  to  prevent  these 
failure  modes  from  developing  tracking  failures.  The  overall  performance  of  the  algorithm  is 
guaranteed  to  be  much  better  than  any  of  the  individual  tracking  algorithms  used  in  the  fusion. 
One  important  aspect  of  the  proposed  algorithm  is  its  recoverability  property  i.e.,  the  ability  to 
recover  following  a  failure  at  a  certain  frame. 

The  flowchart  shown  in  Fig.  5  summarizes  the  basic  building  blocks  of  the  proposed  algorithm. 
In  its  initial  appearance,  the  location  of  the  target  is  determined  from  the  ground  truth  file.  The 
tracking  in  the  subsequent  frames  are  done  automatically  without  any  assistance  from  the  ground 
truth  file. 

2.1  Ego-motion  Compensation  Component 

The  first  component  of  the  proposed  algorithm  is  the  ego-motion  compensation  (EMC). 
Different  and  complementary  ego-motion  compensation  algorithms  are  used  to  generate  initial 
locations  for  the  target.  In  the  flowchart  of  Fig.  5,  we  assume  that  two  EMC  algorithms  are  used 
and  the  two  initial  target  locations  are  referred  to  as  (xi,yi)  and  (x2,  y2),  respectively.  For 
demonstration  purposes,  we  assume  that  the  compensation  is  achieved  by  the  registration  of  the 
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large  white  box  of  the  previous  frame  as  shown  in  Fig.  6.  We  observe  that  EMC  algorithm  #1 
succeeded  in  compensating  the  ego-motion,  while  algorithm  #2  failed.  For  the  subsequent  target 
tracking  to  be  successful,  at  least  one  EMC  algorithm  should  succeed  in  placing  the  target  within 
the  operational  limits  of  the  tracking  algorithm.  The  failure  of  all  EMC  algorithms  will  result  in 
unrecoverable  tracking  error. 


Ego-Motion 

Compensation 

(EMC) 

Initial  target  locations 


Target  Tracking 
(TT) 


Tentative  target 
locations 

Segmented  Targets 


Template 
Matching  (TM) 

Final  target  location 

Final  DetectedTarget 


Update  Target 
Model 


Image  sequence 


initial  frame? 
Next  frameT^ 


Yes 


_ T _ 
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_ i _ 

EMC  4 

algorithm  #1 

^ - 

algorithm  #2 
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Weighted  Composite 
Reference  Function 
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Fig.  5  Flowchart  of  proposed  decision- fusion  algorithm  for  correlation-based  target  tracking  in  FLIR  imagery. 
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Sequence  =  L19NS,  Frame  =  075  Sequence  =  L19NS,  Frame  =  076 


Fig.  6  Example  demonstrating  the  initial  target  locations  generated  by  the  ego-motion  compensation  algorithms, 
(x!,yi)  and  (x2,y2),  and  tentative  target  locations  generated  by  the  tracking  algorithms,  (xn,yn),  (xi2,  yi2),  (x21,y21) 

and  (x22,y22). 


2.2  Target  Tracking  Component 

The  next  component  of  the  proposed  algorithm  is  target  tracking  (TT).  In  the  vicinity  of  the 
initial  locations  generated  by  EMC  algorithms,  different  TT  algorithms  are  used  to  pinpoint  the 
target  by  generating  tentative  locations.  In  the  flowchart  of  Fig.  5,  we  assume  that  two  TT 
algorithms  are  used.  The  number  of  tentative  target  locations  is  the  product  of  the  number  of 
EMC  algorithms  times  the  number  of  TT  algorithms.  Consequently,  in  the  example  of  Fig.  6, 
four  tentative  target  locations,  referred  to  as  (xn,yn),  (X12,  yi2),  (x2i,y2i)  and  (x22,y22),  are 
generated.  The  correct  location  of  the  target  is  (xn,yn)  and  the  other  locations  are  false  alarms. 

2.3  Template  Matching  Component 

This  component  selects  the  final  target  location  from  the  tentative  locations  determined  in  the 
preceding  steps.  This  is  achieved  by  template  matching.  Segmented  targets  (ST;j)  from  the 
tentative  locations  are  compared  with  a  reference  image  for  the  target,  called  the  Weighted 
Composite  Reference  Function  (WCRF),  which  is  discussed  in  next  subsection.  An  Error  Margin 
(EMij)  is  computed  for  every  ST,,,  where  EM,,  reflects  the  dissimilarity  between  ST,,  and  the 
reference  image  WCRF.  The  ST;j  with  the  lowest  EMy  is  selected  as  the  Final  Detected  Target 
(FDT).  In  the  example  of  Fig.  6,  EMu  has  the  lowest  EM.  Therefore,  (xn,yn)  is  selected  as  the 
final  target  location,  and  FDT  is  selected  as  STn. 

2.4  Update  Target  Model 

The  last  component  of  the  proposed  algorithm  is  updating  the  target  model.  A  Weighted 
Composite  Reference  Function  (WCRF)  is  used  to  update  the  target  model.  The  WCRF  is 
generated  by  summing  the  multiplication  of  FDT  images  of  the  target  used  in  the  previous 
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frames  with  a  set  of  weighted  coefficients.  By  allowing  WCRF  to  preserve  its  history,  we 
effectively  alleviate  the  distortion  of  the  reference  image. 

2.5  Data/Decision  Fusion  Results 

The  experiments  performed  on  the  AMCOM  FLIR  data  set  verify  the  robustness  of  the  proposed 
data/decision  fusion  algorithm.  By  using  different  and  complementary  ego-motion  compensation 
and  tracking  algorithms,  we  have  improved  the  chances  of  detecting  the  target.  The  template 
matching  step,  which  selects  the  best  match  to  the  target  model  from  the  possible  locations, 
insures  the  rejection  of  false  alarm  related  errors.  The  WCRF  method  used  for  updating  the  target 
model  alleviates  the  distortion  of  the  reference  image,  and  at  the  same  time,  it  allows  the  target 
model  to  update  its  shape,  size  and  orientation  from  frame  to  frame.  It  also  enhances  the 
probability  of  recovering  after  a  failure  at  a  certain  frame.  The  multiple  target  tracking  results 
obtained  using  the  data/decision  fusion  algorithm  are  shown  in  Appendix  C.  Summarized  fusion 
results  are  also  shown  in  Appendix  D.  In  the  first  experiment  we  fused  CC-IVF-TM  and  FJTC- 
MSDF-FJTC  tracking  algorithms.  The  fusion  algorithms  performed  better  than  the  individual 
algorithms  used  in  the  fusion.  In  the  second  experiment,  we  obtained  better  results  by  fusing  four 
algorithms:  the  two  algorithms  used  in  the  first  experiment  plus  two  additional  algorithms,  FJTC- 
IVF-TM  and  CC-FJTC-TM. 


3.  Performance  Metrics 

FLIR  image  sequences  are  very  complex  in  terms  of  the  target  signature  and  background,  which 
makes  the  tracking  very  difficult.  We  observed  that  the  input  data  complexity  directly  affects  the 
performance  of  various  tracking  algorithms.  On  the  other  hand,  the  performance  of  the  tracking 
algorithms  can  be  evaluated  by  analyzing  the  output  results. 

3.1  Effect  of  Data  on  the  Performance  of  an  Algorithm 

Input  data  complexity  directly  influences  the  amount  of  useful  information  available  in  the  target 
signature.  We  developed  two  new  performance  metrics  to  measure  the  effect  of  input  data  on  the 
performance  of  the  target  tracking  algorithms  which  include  local  intensity  ratio  and  local 
contrast  ratio.  The  local  intensity  ratio  (LIR)  corresponds  to  the  intensity  variation  between  the 
target  and  the  neighboring  background  in  the  infrared  input  image.  The  local  contrast  ratio 
(LCR)  corresponds  to  the  contrast  variation  between  the  target  and  the  neighboring  background 
in  the  infrared  input  image.  Fig.  7  shows  the  data  complexity  in  terms  of  LIR  and  LCR  for  all 
frames  of  a  test  sequence  (L1415),  which  contains  a  single  target  of  interest.  Fig.  8  shows  the 
same  parameters  for  a  multi-target  image  sequence  (L1817S1). 
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Frame  Number 

(d) 

Fig.  7  LIR  and  LCR  results  for  sequence  (L1415),  a)  first  frame,  b)  last  frame,  c)  LIR  result,  and  d)  LCR  result  with 

100  %  tracking. 


After  testing  the  LIR  and  LCR  metrics  with  all  sequences,  we  found  a  threshold  value  of  1.05  for 
LIR  and  1.0  for  LCR.  If  LIR  is  above  1.05  and  LCR  is  above  1,  it  is  highly  likely  that  a  tracking 
algorithm  will  successfully  track  the  target.  If  LIR  is  close  to  1.05  and  LCR  is  close  to  1,  a 
tracking  algorithm  may  or  may  not  detect  and  track  the  target.  If  the  LCR  is  below  1.05  and  LCR 
is  below  1 ,  it  is  highly  likely  that  a  tracking  algorithm  will  fail  to  detect  and  track  the  target. 
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Fig.  8  LIR  and  LCR  results  for  sequence  (L1817S1):  a)  first  frame,  b)  LIR  result,  c)  frame  #  63  ,  and  d)  LCR  result. 


3.2  Metrics  for  Evaluating  Performance  of  a  Tracking  Algorithm 

The  similarity  between  a  target  and  background,  blended  target,  rotation  and  scale  variations  of 
the  target,  noise  and  clutter  present  in  FLIR  images  pose  various  challenges  in  target  detection 
and  tracking.  We  investigated  the  performance  of  the  tracking  algorithms  using  various 
performance  metrics  for  real  life  FLIR  image  sequences.  We  choose  three  metrics  to  evaluate  the 
tracking  algorithms  which  are  peak-to-sidelobe  ratio  (PSR),  peak-to-correlation  energy  (PCE) 
and  Fisher  ratio  (FR)  [4,5].  These  performance  metrics  are  used  to  estimate  the  reliability  of  the 
tracking  algorithm.  For  illustration  purposes,  we  report  herein  the  performance  of  the  CC-FJTC- 
TM  algorithm  using  the  aforementioned  metrics. 

The  PSR  determines  the  sharpness  of  the  peak  by  comparing  the  highest  peak  intensity  with  a 
side-lobe  peak  intensity.  The  PCE  calculates  the  ratio  between  the  correlation  peak  energy  and 
the  total  energy  of  the  correlation  plane.  The  higher  the  value  of  PCE,  the  sharper  the  correlation 
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peak.  Fisher  ratio  is  used  to  measure  the  performance  of  an  algorithm  with  respect  to  the 
distortions  associated  with  a  target.  It  is  utilized  to  evaluate  the  performance  of  an  algorithm  for 
distortion  invariant  target  detection.  When  the  FR  is  close  to  zero,  it  indicates  that  the  target  in 
the  previous  and  current  frames  is  same.  When  FR  is  higher  than  zero,  it  implies  that  the  targets 
are  different  or  there  are  significant  variations  between  the  target  in  the  previous  frame  and  the 
current  frame.  Figures  9,  10  and  11  show  the  performance  results  of  CC-FJTC-TM  for  two 
sequences  (L1415  and  LI 871  SI),  respectively. 

To  determine  the  thresholds  for  each  of  these  performance  metrics,  we  considered  the  tracking 
results  for  all  sequences  and  determined  the  threshold  for  PSR,  PCE  and  FR  as  15,  0.002  and  0.5, 
respectively.  For  the  first  sequence  (L1415),  PSR  and  PCE  are  above  their  threshold  and  FR  is 
below  the  threshold  for  all  frames  as  shown  in  Figs.  9(a),  9(b)  and  9(c),  respectively.  Therefore, 
the  CC-FJTC-TM  algorithm  tracks  the  target  in  this  sequence  with  100%  accuracy. 


L1415  PSR 


(a) 


LI  4 1  5  ROE 
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LI  41  5  FFR 


(c) 

Fig.  9  (a)  PSR  for  L1415,  (b)  PCE  for  L1415  and  (c)  FR  for  L1415. 


(a) 


(b) 


(c) 

Fig.  10  (a)  PSR  results  for  target-1  in  sequence  L1817S1,  (b)  PCE  results,  and  (c)  FR  results. 
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(a) 


(b) 


0.5 
□  45 
□ .  4-  - 
0.35 
0.3  - 
0  2 
0.2  - 
O  .  1  5 
0.1  - 
0.05 


(c) 

Fig.  1 1  (a)  PSR  results  for  target-2  in  sequence  L1817S1,  (b)  PCE  results,  and  (c)  FR  results. 


The  performances  metrics  for  the  second  sequence  (L1817S1)  is  shown  in  Figs.  10  and  11.  From 
Fig.  10,  it  is  evident  that  the  PSR  and  PCE  values  for  Target  #1  change  dramatically  i.e.,  the 
values  drop  lower  than  the  threshold  at  around  frame  number  60.  This  implies  that  this  algorithm 
fails  to  track  Target  #1  at  around  frame  number  60.  If  we  consider  Target  #2  (Figure  1 1)  in  the 
same  sequence,  it  is  evident  that  the  PCE  and  PSR  values  are  above  the  threshold  and  FR  is  close 
to  0,  implying  that  the  algorithm  can  detect  and  track  Target  #2  with  100%  accuracy. 


4.  Target  Detection 

We  investigated  two  algorithms  for  target  detection  in  the  initial  frame  of  a  sequence  using  (i) 
multilevel  data  fusion  technique  assuming  no  a  priori  information  is  available  about  the  target, 
and  (ii)  Law’s  Texture  Energy  Measures  assuming  the  number  of  targets  present  in  the  image  are 
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known  a  priori.  The  developed  algorithms  utilize  target  intensity,  contrast,  as  well  as  other 
properties  to  detect  brighter  hot  spots  which  correspond  to  possible  targets. 

4.1  Target  Detection  Using  Multilevel  Data  Fusion 

This  technique  was  originally  introduced  by  Borghys  et  al.  [6].  It  is  based  on  constructing  co¬ 
occurrence  matrix  locally  for  the  entire  input  image  to  calculate  features  such  as  energy,  contrast, 
maximum  probability,  entropy,  homogeneity  and  variance.  These  features  are  then  fused  together 
based  on  their  coefficients  obtained  through  training.  However,  for  some  cases,  we  found  that  the 
final  output  may  split  a  probable  single  target  into  two  parts  if  the  existing  technique  is  used. 
This  may  complicate  the  target  detection  process.  To  eliminate  this  limitation,  an  average 
filtering  approach  is  used  for  the  final  output  to  join  the  small  pieces  that  appear  closer  to  each 
other.  This  enhances  the  detection  accuracy  and  eliminates  false  alarms.  Thereafter,  a  threshold 
operation  is  applied  to  reject  unwanted  regions.  Two  examples  of  target  detection  in  the  first 
frame  using  the  above  mentioned  technique  are  depicted  in  Figs.  12  and  13,  respectively. 


Fused  image  Processed  image 


10  20  30  40  50  60  70  80  90  100  110  10  20  30  40  50  60  70  80  90  100  110 


Thresholded  image 
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20  40  60  80  100  120 

Fig.  12  Target  detection  using  multilevel  data  fusion  in  sequence  L1608 
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Fused  image  Processed  image 


20  40  60  80  100  120  20  40  60  80  100  120 


Fig.  13.  Target  detection  using  multilevel  data  fusion  in  sequence  L2312. 

This  technique  shows  some  promising  results  but  it  requires  a  lot  of  computation  as  a  co¬ 
occurrence  matrix  needs  to  be  constructed  for  every  sub-region  of  the  image.  After  further 
testing,  we  obtained  similar  results  by  applying  a  combination  of  a  standard  deviation  filter  and 
an  average  filter  to  the  input  image.  This  technique  requires  less  computation  time  and  does  not 
need  any  training  when  compared  to  the  previous  technique.  Figures  14  and  15  show  the  results 
obtained  for  the  same  sequence  using  our  enhanced  algorithm. 


Standard  deviation  filter 


10  20  30  40  50  60  70  80  90  100  110 


Standard  deviation  filter 


10  20  30  40  50  60  70  80  90  100  110 
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threshold  operation 


10  20  30  40  50  60  70  80  90  100  110 


Detected  targets 


20  40  60  80  100  120 


Fig.  14.  Target  detection  using  standard  deviation  and  multilevel  data  fusion  in  sequence  L1608. 


4.2  Target  Detection  Using  Law’s  Texture  Energy  Measures 

In  this  work,  to  detect  targets  in  the  initial  frame,  morphological  operators  and  Law’s  texture 
energy  measures  [7-10]  are  used.  Before  starting  any  processing  on  the  images,  the  gray  levels  of 
each  image  are  rescaled  between  0  to  255.  At  first,  the  following  steps  are  followed  to  find  the 
local  maxima: 


19 


1)  Morphological  dilation  with  a  centered  horizontal  element  H  of  width  3  is  applied  so  that 
small  objects  become  more  perceptible. 

2)  Difference  between  dilated  image  and  original  image  yields  the  edge. 

3)  Addition  of  the  original  image  with  the  image  obtained  from  step  2,  generates  an  enhanced 
version  of  the  original  image. 

4)  For  further  enhancement,  the  image  from  step  3  is  again  dilated  with  the  same  horizontal 
element  H  of  width  3. 

5)  Subtraction  of  the  original  image  from  the  image  at  step  5  yields  broad  or  wide  edges. 

6)  Average  filtering  of  the  image  at  step  5  fills  the  blank  space  enclosed  by  the  edge  of  any 
object. 

7)  A  threshold  is  then  applied  to  the  image  at  step  6  to  eliminate  unwanted  smaller  regions. 

8)  After  the  application  of  threshold,  average  filtering  is  reapplied  to  connect  adjacent  parts  of 
an  object  if  there  are  any. 

The  threshold  value  is  chosen  adaptively  depending  on  the  ratio  between  the  maximum  intensity 

of  an  image  and  mean  intensity  of  the  same  image.  The  resulting  images,  obtained  after  each  of 

the  above  steps,  are  shown  in  Figs.  16  through  23,  respectively. 


Fig.  16.  Original  rescaled  Fig.  17.  Output  of  step  1.  Fig.  18  Output  of  step  2.  Fig.  19  Output  of  step  3. 
image. 


Fig.  20  Output  of  step  4.  Fig.  21  Output  of  step  5.  Fig.  22  Output  of  step  6.  Fig.  23  Output  of  step  8. 

Thereafter,  Law’s  texture  energy  measures  are  applied  to  find  the  possible  targets  among  the 
local  maxima  shown  in  Fig.  23.  The  target  detection  algorithm  then  searches  for  any  non-zero 
values  in  the  image  resulting  from  step  8.  All  adjacent  non-zero  pixels  are  considered  as  a  single 
object.  After  applying  the  search  operation,  the  resulting  detected  regions  containing  both 
potential  targets  and  non-target  objects  are  obtained  as  shown  in  Fig.  24. 

Now  the  challenge  is  to  discriminate  true  targets  from  non-target  objects.  Texture  may  be  used  to 
distinguish  between  true  targets  or  target  like  background  in  FLIR  Imagery.  Law’s  texture 
energy  measures  are  basically  feature  extraction  schemes  based  on  gradient  operators.  This 
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scheme  uses  5  masks  which  are  obtained  by  convolving  five  1-dimentional  vectors,  each 
representing  level,  edge,  spot,  wave  and  ripple,  respectively,  as  shown  below: 

•  Level  L5  =  [  1  4  6  4  1] 

•  Edge  E5  =  [-1  -2  0  2  1] 

•  Spot  S5  =  [-1  0  2  0  -1] 

•  Wave  W5=  [-1  2  0-2  1] 

•  Ripple  R5=  [1-4  6-4  1] 

The  following  5  masks  are  then  applied  to  the  image  to  eliminate  the  false  positives  from  Fig.  24. 
The  five  output  images  of  the  five  masks  are  shown  in  Figs.  25  to  29,  respectively. 
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Fig.  24  Candidate  targets’  Fig.  25  Output  of  mask  (E5')L5  Fig.  26  Output  of  mask  (R5')L5 

positions 


Fig.  27  Output  of  mask  (S5')E5  Fig.  28  Output  of  mask  (S5')S5  Fig.  29  Output  of  mask  (R5')R5 
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After  calculating  the  above  features  for  the  detected  objects’  region,  true  targets  are  separated 
from  non-target  objects.  The  final  detection  result  of  the  true  targets,  after  eliminating  the  false 
target  like  background  using  Law’s  Texture  Energy  Measures,  is  shown  in  Fig.  30.  It  is  found 
that  the  texture  feature  values  are  much  higher  for  potential  targets  than  non-targets.  The  results 
of  this  algorithm  for  target  detection  in  the  frame,  where  the  target  appears  first,  are  shown  in 
Appendix  E. 


Fig.  30  Detected  true  targets  using  Law’s  Texture  Energy  Measures. 


5.  Conclusion 

In  this  report,  we  presented  an  overview  of  the  works  that  was  performed  during  the  stated 
period.  The  USA  segment  of  the  research  work  lead  to  the  development  of  four  target  tracking 
algorithms  and  two  data/decision  fusion  algorithms.  The  performance  of  these  algorithms  has 
been  tested  using  real  life  FLIR  image  sequences  supplied  by  AMCOM.  The  publications 
resulted  from  this  project  are  listed  in  Appendix  F. 

The  main  characteristic  of  the  FJTC-IVF-TM  algorithm  is  that  it  uses  target  intensity  and  shape 
information  for  detection  and  tracking.  Normalized  cross-correlation  techniques  (CC-IYF-TM 
and  CC-FJTC-TM)  further  enhances  the  robustness  of  the  tracking  algorithm.  The  modified  SDF 
based  invariant  FJTC  (FJTC-MSDF-FJTC)  target  tracking  algorithm  efficiently  utilizes  scale, 
rotation  as  well  as  neighborhood  information  for  efficient  target  tracking.  The  data/decision 
fusion  algorithm  combines  two  or  more  of  the  above  algorithms  to  ensure  much  better  detection 
and  tracking  performance. 

The  impact  of  input  data  complexity  and  correlation  output  results  are  analyzed  and  metrics  are 
developed  to  quantify  the  tracking  performance  for  both  input  data  (image  sequences)  and  results 
generated  by  the  various  algorithms.  In  the  above  mentioned  tracking  techniques,  we  assume  that 
the  target  is  known  in  the  first  frame.  To  detect  the  target  in  the  first  frame  without  any  a  priori 
knowledge  is  a  challenging  task.  In  this  project,  we  also  investigated  two  algorithms  for 
detecting  target(s)  in  the  initial  frame  of  a  FFIR  sequence. 
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Appendix  A:  Target  Detection  and  Tracking  Results  Generated  by  the 
Algorithms  Developed  at  USA 

Algorithm  #1:  FJTC-IVF-TM  algorithm 
Algorithm  #2:  CC-IVF-TM  algorithm 
Algorithm  #3:  CC-FJTC-TM  algorithm 
Algorithm  #4:  FJTC-MSDF-FJTC  algorithm 


Seq  no. 

Sequence  name 

Total  frames 

Targets 

Tracking  Results 

Alg#l 

Alg#2 

Alg#3 

Alg#4 

1 

L14_15 

281 

tar  1  -  mantrack 

100% 

100% 

100% 

100% 

2 

L1520 

215 

tar  1  -  tank 

100% 

100% 

100% 

100% 

3 

L15NS 

320 

tar  1  -  tankl 

100% 

6% 

100% 

100% 

tar  2  -  truck  1 

25% 

65% 

100% 

100% 

4 

L1604 

400 

tar  1  -  tankl 

100% 

0% 

100% 

100% 

5 

L16_07 

410 

tar  1  -  tankl 

0% 

0% 

25% 

2% 

tar  2  -  mantrack 

100% 

45% 

100% 

100% 

6 

L1608 

290 

tar  1  -  m60 

100% 

100% 

tar  2  -  apcl 

100% 

100% 

tar  3  -  mantrack? 

76% 

EH 

76% 

7 

L16_18 

300 

tar  1  -  m60 

100% 

100% 

100% 

100% 

tar  2  -  ape 

100% 

100% 

100% 

100% 

tar  3  -  track 

100% 

100% 

100% 

100% 

8 

L17_01 

388 

tar  1  -  bradley 

100% 

100% 

100% 

100% 

tar  2  -  pickup 

100% 

100% 

100% 

100% 

9 

L17_02 

698 

tar  1  -  mantrack 

45% 

100% 

100% 

100% 

tar  2  -  pickup 

20% 

100% 

100% 

100% 

tar  3  -  bradley 

20% 

2% 

50% 

67% 

tar  4  -  mantrack 

0% 

0% 

100% 

1% 

tar  5  -  tank 

0% 

100% 

100% 

100% 

10 

L17_20 

778 

tar  1  -  target 

0% 

100% 

100% 

0% 

tar  2  -  m60 

0% 

0% 

100% 

100% 

11 

L18_03 

450 

tar  1  -  bradley 

1% 

1% 

100% 

100% 

12 

L18_05 

779 

tar  1  -  tankl 

100% 

50% 

100% 

100% 

tar  2  -  apcl 

60% 

100% 

100% 

100% 

tar  3  -  m60 

0% 

0% 

1% 

2% 

tar  4  -  tankl 

0% 

0% 

1% 

100% 

13 

L18_07 

260 

tar  1  -  bradley 

10% 

7% 

100% 

100% 

14 

L18_12 

300 

tar  1  -  tankl 

80% 

100% 

100% 

53% 

tar  2  -  bradley 

100% 

100% 

100% 

100% 

tar  3  -  m60 

100% 

100% 

100% 

100% 

15 

L 1 8_ 1 3 

326 

tar  1  -  tankl 

100% 

80% 

100% 

100% 

tar  2  -  m60 

100% 

100% 

100% 

100% 

tar  3  -  ape 

100% 

100% 

100% 

100% 

16 

L 1 8_  1 5  Seq  1 

339 

tar  1  -  m60 

50% 

1% 

100% 

100% 

tar  2  -  track 

100% 

100% 

100% 

100% 

tar  3  -  bradley 

0% 

0% 

100% 

10% 

23 


tar  4  -  tankl 

0% 

0% 

100% 

100% 

17 

L 1 8_  1 5  Seq  2 

342 

tar  1  -  tankl 

30% 

30% 

100% 

100% 

tar  2  -  bradley 

50% 

0% 

100% 

4% 

tar  3  -  m60 

90% 

0% 

100% 

100% 

18 

L1816 

328 

tar  1  -  tankl 

100% 

16% 

tar  2  -  m60 

100% 

1% 

19 

L18_17  Seq  1 

240 

tar  1  -  tankl 

50% 

25% 

50% 

tar  2  -  m60 

100% 

100% 

100% 

20 

L18_17  Seq  2 

266 

tar  1  -  tankl 

50% 

25% 

50% 

I1H 

tar  2  -  m60 

100% 

100% 

100% 

21 

L1818 

365 

tar  1  -  mantrack 

20% 

50% 

50% 

37% 

tar  2  -  target 

10% 

100% 

100% 

100% 

tar  3  -  apcl 

10% 

50% 

100% 

100% 

tar  4  -  m60 

10% 

0% 

10% 

10% 

tar  5  -  tankl 

50% 

0% 

1% 

100% 

tar  6  -  test  van 

40% 

100% 

100% 

100% 

22 

L1901 

240 

tar  1  -  tankl 

1% 

5% 

1% 

100% 

23 

L1902 

270 

tar  1  -  tankl 

0% 

1% 

1% 

100% 

24 

L1904 

270 

tar  1  -  tankl 

90% 

100% 

100% 

37% 

25 

L1906 

265 

tar  1  -  tankl 

5% 

5% 

1% 

100% 

tar  2  -  ape 

5% 

5% 

1% 

0% 

tar  3  -  mantrack 

100% 

100% 

100% 

100% 

tar  4  -  van 

100% 

100% 

100% 

100% 

26 

L1907 

195 

tar  1  -  mantrack 

0% 

0% 

100% 

100% 

27 

L1910 

130 

tar  1  -  tankl 

100% 

100% 

100% 

100% 

tar  2  -  ape 

100% 

100% 

100% 

100% 

28 

L1911 

165 

tar  1  -  tankl 

100% 

100% 

1% 

100% 

tar  2  -  ape 

100% 

100% 

100% 

100% 

29 

L1913 

265 

tar  1  -  tankl 

95% 

100% 

70% 

100% 

tar  2  -  ape 

95% 

100% 

100% 

100% 

tar  3  -  m60 

100% 

100% 

100% 

100% 

tar  4  -  mantrack 

95% 

100% 

100% 

100% 

30 

350 

tar  1  -  tankl 

10% 

0% 

tar  2  -  apcl 

95% 

96% 

3% 

tar  3  -  m60 

10% 

0% 

1% 

tar  4  -  van 

50% 

0% 

2% 

31 

L1918 

260 

tar  1  -  tankl 

20% 

10% 

100% 

100% 

tar  2  -  m60 

100% 

100% 

100% 

100% 

32 

L19NS 

275 

tar  1  -  m60 

50% 

100% 

60% 

100% 

tar  2  -  mantrack 

100% 

100% 

100% 

100% 

tar  3  -  tankl 

90% 

2% 

100% 

25% 

33 

L20  04 

368 

tar  1  -  apcl 

100% 

100% 

100% 

100% 

34 

L20  08 

348 

tar  1  -  apcl 

10% 

5% 

100% 

100% 

35 

L20_17 

308 

tar  1  -  tankl 

3% 

2% 

100% 

100% 

36 

L2018 

448 

tar  1  -  tankl 

100% 

100% 

100% 

100% 

37 

L20  20 

420 

tar  1  -  tankl 

10% 

tar  2  -  target 

20% 

38 

L2104 

760 

tar  1  -  bradley 

0% 

28% 

100% 

36% 

24 


tar  2  -  tankl 

0% 

3% 

10% 

1% 

39 

L2115 

738 

tar  1  -  bradley 

10% 

3% 

5% 

1% 

40 

L2117 

360 

tar  1  -  apcl 

0% 

100% 

100% 

1% 

41 

L22  06 

348 

tar  1  -  tankl 

40% 

35% 

100% 

11% 

42 

L22-08 

380 

tar  1  -  apcl 

0% 

0% 

100% 

1% 

43 

L2210 

398 

100% 

100% 

100% 

100% 

44 

L22_14 

350 

tar  1  -  m60 

5% 

0% 

100% 

tar  2  -  target 

0% 

0% 

100% 

45 

L23_12 

368 

tar  1  -  apcl 

100% 

100% 

100% 

100% 

46 

M1406 

380 

tar  1  -  bradley 

100% 

100% 

100% 

100% 

47 

M14_07 

400 

tar  1  -  bradley 

100% 

100% 

100% 

100% 

48 

M1410 

498 

tar  1-  tankl 

100% 

100% 

tar  2  -  mantruck 

80% 

98% 

79% 

49 

M1413 

380 

tar  1  -  mantruck 

100% 

100% 

100% 

100% 

50 

M14_15 

528 

tar  1  -  mantruck 

0% 

100% 

70% 

70% 

tar  2  -  target 

0% 

100% 

1% 

33% 

Appendix  B:  Comparative  performance  evaluation  of  the  four  tracking  algorithms 

developed  at  USA 

Total  number  of  sequences 

50 

Total  number  of  multiple  targets  sequences 

28 

Total  number  of  reappearing  targets 

16 

Total  number  of  targets  in  all  50  sequences 

101 

Tracking  Algorithm  Type 

FJTC- 

CC- 

CC- 

FJTC- 

IVF- 

IVF- 

FJTC- 

MSDF- 

TM 

TM 

TM 

FJTC 

Total  number  of  successful  tracked  sequences: 

15 

18 

31 

23 

Total  number  of  successful  tracked  targets  excluding  reappearance: 

38 

50 

75 

65 

Total  percentage  of  successful  tracking  excluding  reappearance: 

55% 

58% 

82% 

72% 

Total  number  of  targets'  tracking  failure  due  to  overlapping  : 

3 

3 

3 

3 

Total  number  of  targets'  tracking  failure  due  to  camera  noise  : 

7 

6 

3 

3 

Total  number  of  targets'  tracking  failure  due  to  smoke  : 

24 

24 

3 

20 

Total  number  of  targets'  tracking  failure  due  to  scaling  : 

0 

0 

0 

1 

Total  number  of  targets'  tracking  failure  due  to  distraction  by  vicinity  object 

3 

3 

3 

5 

Total  number  of  targets'  tracking  failure  due  to  other  reasons  : 

26 

15 

14 

4 

25 


Appendix  C:  Target  Detection  and  Tracking  Results  Generated  by  the  Data/ 
Decision  Fusion  Algorithms  Developed  at  USA 


Algorithm  #5  (DF  #1):  2  USA  algorithms  (CC-IVF-TM  and  FJTC-MSDF-FJTC) 

Algorithm  #6  (DF  #2):  4  USA  algorithms  (CC-IVF-TM,  FJTC-IVF-TM,  CC-FJTC-TM,  FJTC-MSDF-FJTC) 


Seq 

no. 

Sequence 

name 

Total 

frames 

Targets 

Tracking 
Results 
(DF  #1) 

Tracking 
Results 
(DF  #2) 

1 

L1415 

281 

tar  1  -  mantruck 

100% 

100% 

2 

L1520 

215 

tar  1  -  tank 

100% 

100% 

3 

L15_NS 

320 

tar  1  -  tankl 

100% 

100% 

tar  2  -  truck  1 

100% 

100% 

4 

L1604 

400 

tar  1  -  tankl 

100% 

100% 

5 

L1607 

410 

tar  1  -  tankl 

10% 

30% 

tar  2  -  mantruck 

100% 

100% 

6 

L1608 

290 

tar  1  -  m60 

100% 

100% 

tar  2  -  apcl 

100% 

100% 

tar  3  -  mantruck? 

100% 

100% 

7 

L1618 

300 

tar  1  -  m60 

100% 

100% 

tar  2  -  ape 

100% 

100% 

tar  3  -  truck 

100% 

100% 

8 

L1701 

388 

tar  1  -  bradley 

100% 

100% 

tar  2  -  pickup 

100% 

100% 

9 

L1702 

698 

tar  1  -  mantruck 

100% 

100% 

tar  2  -  pickup 

100% 

100% 

tar  3  -  bradley 

18% 

41% 

tar  4  -  mantruck 

0% 

55% 

tar  5  -  tank 

100% 

100% 

10 

L1720 

778 

tar  1  -  target 

100% 

100% 

tar  2  -  m60 

100% 

100% 

11 

L1803 

450 

tar  1  -  bradley 

100% 

70% 

12 

L1805 

779 

tar  1  -  tankl 

100% 

100% 

26 


tar  2  -  apcl 

100% 

100% 

tar  3  -  m60 

100% 

100% 

tar  4  -  tankl 

100% 

100% 

13 

L18_07 

260 

tar  1  -  bradley 

100% 

100% 

14 

L18_12 

300 

tar  1  -  tankl 

100% 

100% 

tar  2  -  bradley 

100% 

100% 

tar  3  -  m60 

100% 

100% 

15 

LI  81 3 

326 

tar  1  -  tankl 

100% 

100% 

tar  2  -  m60 

100% 

100% 

tar  3  -  ape 

100% 

100% 

16 

LI  8 _ 1 5 

Seq  1 

339 

tar  1  -  m60 

100% 

100% 

tar  2  -  truck 

100% 

100% 

tar  3  -  bradley 

100% 

100% 

tar  4  -  tankl 

35% 

35% 

17 

L18  15 
Seq  2 

342 

tar  1  -  tankl 

100% 

100% 

tar  2  -  bradley 

35% 

100% 

tar  3  -  m60 

100% 

100% 

18 

L1816 

328 

tar  1  -  tankl 

100% 

100% 

tar  2  -  m60 

100% 

100% 

19 

L1817 
Seq  1 

240 

tar  1  -  tankl 

100% 

100% 

tar  2  -  m60 

100% 

100% 

20 

L18_17 
Seq  2 

266 

tar  1  -  tankl 

100% 

100% 

tar  2  -  m60 

100% 

100% 

21 

LI  81 8 

365 

tar  1  -  mantrack 

50% 

100% 

tar  2  -  target 

100% 

100% 

tar  3  -  apcl 

100% 

50% 

tar  4  -  m60 

10% 

25% 

tar  5  -  tankl 

1% 

1% 

tar  6  -  testvan 

100% 

100% 

22 

L1901 

240 

tar  1  -  tankl 

100% 

100% 

23 

L1902 

270 

tar  1  -  tankl 

100% 

100% 

24 

L1904 

270 

tar  1  -  tankl 

100% 

100% 

27 


25 

L1906 

265 

tar  1  -  tankl 

100% 

85% 

tar  2  -  ape 

100% 

100% 

tar  3  -  mantruck 

100% 

100% 

tar  4  -  van 

100% 

100% 

26 

L1907 

195 

tar  1  -  mantruck 

100% 

100% 

27 

L1910 

130 

tar  1  -  tankl 

100% 

100% 

tar  2  -  ape 

100% 

100% 

28 

L19 _ 1 1 

165 

tar  1  -  tankl 

100% 

100% 

tar  2  -  ape 

100% 

100% 

29 

L19 _ 13 

265 

tar  1  -  tankl 

100% 

100% 

tar  2  -  ape 

100% 

100% 

tar  3  -  m60 

100% 

100% 

tar  4  -  mantruck 

100% 

100% 

30 

L19 _ 1 5 

350 

tar  1  -  tankl 

45% 

100% 

tar  2  -  apcl 

100% 

100% 

tar  3  -  m60 

100% 

100% 

tar  4  -  van 

100% 

100% 

31 

L19 _ 1 8 

260 

tar  1  -  tankl 

100% 

100% 

tar  2  -  m60 

0% 

0% 

32 

L19NS 

275 

tar  1  -  m60 

100% 

100% 

tar  2  -  mantruck 

100% 

100% 

tar  3  -  tankl 

3% 

100% 

33 

L20  04 

368 

tar  1  -  apcl 

100% 

100% 

34 

L20  08 

348 

tar  1  -  apcl 

100% 

100% 

35 

L2017 

308 

tar  1  -  tankl 

100% 

100% 

36 

L2018 

448 

tar  1  -  tankl 

100% 

100% 

37 

L20  20 

420 

tar  1  -  tankl 

100% 

100% 

tar  2  -  target 

89% 

95% 

38 

L2104 

760 

tar  1  -  bradley 

100% 

100% 

tar  2  -  tankl 

100% 

100% 

39 

L2115 

738 

tar  1  -  bradley 

100% 

100% 

40 

L2117 

360 

tar  1  -  apcl 

100% 

100% 

28 


41 

L22  06 

348 

tar  1  -  tankl 

100% 

100% 

42 

L22-08 

380 

tar  1  -  apcl 

100% 

100% 

43 

L2210 

398 

100% 

100% 

44 

L22_14 

350 

tar  1  -  m60 

18% 

100% 

tar  2  -  target 

100% 

100% 

45 

L23_12 

368 

tar  1  -  apcl 

100% 

100% 

46 

M1406 

380 

tar  1  -  bradley 

100% 

100% 

47 

M1407 

400 

tar  1  -  bradley 

100% 

100% 

48 

M1410 

498 

tar  1-  tankl 

100% 

100% 

tar  2  -  mantruck 

100% 

100% 

49 

M1413 

380 

tar  1  -  mantruck 

100% 

100% 

50 

M1415 

528 

tar  1  -  mantruck 

100% 

100% 

tar  2  -  target 

100% 

100% 

Appendix  D:  Performance  evaluation  of  data/decision  fusion 

algorithms 

Algorithm  #5  (DF  #1):  2  USA  algorithms  (CC-IVF-TM  and  FJTC-MSDF-FJTC) 

Algorithm  #6  (DF  #2):  4  USA  algorithms  (CC-IVF-TM,  FJTC-IVF-TM,  CC-FJTC-TM, 
FJTC-MSDF-FJTC) 

Total  number  of  sequences  : 

50 

Total  number  of  multiple  targets  sequences: 

28 

Total  number  of  reappearing  targets  : 

16 

Total  number  of  targets  : 

101 

DF  #  1 

DF  #  2 

Total  number  of  successful  tracked  sequences: 

31 

31 

Total  number  of  successful  tracked  targets  excluding  reappearance: 

89 

93 

Total  percentage  of  successful  tracking  excluding  reappearance: 

91% 

94% 

Total  number  of  targets'  tracking  failure  due  to  overlapping  : 

2 

2 

Total  number  of  targets'  tracking  failure  due  to  camera  noise  : 

2 

2 

Total  number  of  targets'  tracking  failure  due  to  smoke  : 

2 

2 

Total  number  of  targets'  tracking  failure  due  to  scaling  : 

0 

0 

Total  number  of  targets'  tracking  failure  due  to  distraction  by 
vicinity  object 

2 

1 

Total  number  of  targets'  tracking  failure  due  to  other  reasons  : 

4 

1 

29 


Appendix  E:  Target  Detection  at  the  Frame  of  First  Appearance 


Total  number  of  sequences  : 

Total  number  of  multiple  targets  sequences: 

Total  number  of  frames  where  target  appears  for  the  first  time: 
Total  number  of  targets  : 

Total  number  of  successful  detected  targets: 

Total  number  of  targets  which  cannot  be  discriminated: 

Total  number  of  targets  which  cannot  be  detected: 

Total  percentage  of  successful  detection: 


38/101= 


Targets’  names 


L14  15  281  tar  1  -  mantruck 


Frame 
number  at 
which 
target 
appears 
for  the 
first  time 


Can  detect 
and 

discriminate 

from 

nontargets 


Detection  Results 

Can  detect  but 
fail  to 

discriminate 

from 

nontargets 


Can  not 
detect 


L15  20  215 


tar  1  -  tank 


4  L16  04  400  tar  1  -  tankl 


yes 


10 

L1720 

778 

tar  1  -  target 

1 

yes 

tar  2  -  m60 

44 

yes 

11 

L1803 

450 

tar  1  -  bradley 

1 

yes 

12 

L1805 

779 

tar  1  -  tankl 

1 

yes 

tar  2  -  apcl 

1 

yes 

tar  3  -  m60 

441 

yes 

tar  4  -  tankl 

696 

yes 

13 

L1807 

260 

tar  1  -  bradley 

1 

yes 

14 

LI  8  12 

300 

tar  1  -  tankl 

1 

yes 

tar  2  -  bradley 

1 

yes 

tar  3  -  m60 

67 

yes 

15 

L1813 

326 

tar  1  -  tankl 

1 

yes 

tar  2  -  m60 

1 

yes 

tar  3  -  ape 

1 

yes 

16 

L18  15 
Seq  1 

339 

tar  1  -  m60 

1 

yes 

tar  2  -  truck 

11 

yes 

tar  3  -  tankl 

51 

yes 

tar  4  -  bradley 

51 

yes 

17 

L1815 
Seq  2 

342 

tar  1  -  m60 

1 

yes 

tar  2  -  tankl 

36 

yes 

tar  3  -  bradley 

42 

yes 

18 

L1816 

328 

tar  1  -  tankl 

1 

yes 

tar  2  -  m60 

1 

yes 

19 

L18_17 
Seq  1 

240 

tar  1  -  tankl 

1 

yes 

tar  2  -  m60 

1 

yes 

20 

L18_17 
Seq  2 

266 

tar  1  -  tankl 

1 

yes 

tar  2  -  m60 

1 

yes 

21 

L18_18 

365 

tar  1  -  mantruck 

1 

yes 

tar  2  -  target 

3 

yes 

tar  3  -  apcl 

22 

yes 

tar  4  -  m60 

82 

yes 

tar  5  -  tankl 

152 

yes 

tar  6  -  testvan 

362 

yes 

22 

L1901 

240 

tar  1  -  tankl 

1 

yes 

23 

L1902 

270 

tar  1  -  tankl 

1 

yes 

24 

L1904 

270 

tar  1  -  tankl 

1 

yes 

25 

L1906 

265 

tar  1  -  ape 

1 

yes 

tar  2  -  tankl 

1 

yes 

tar  3  -  mantruck 

1 

yes 

tar  4  -  van 

55 

yes 

31 


26 

L1907 

195 

tar  1  -  mantruck 

120 

yes 

27 

LI  91 0 

130 

tar  1  -  tankl 

17 

yes 

tar  2  -  ape 

57 

yes 

28 

L1911 

165 

tar  1  -  tankl 

1 

yes 

tar  2  -  ape 

1 

yes 

29 

L1913 

265 

tar  1  -  tankl 

1 

yes 

tar  2  -  ape 

1 

yes 

tar  3  -  m60 

178 

yes 

tar  4  -  mantruck 

232 

yes 

30 

L1915 

350 

tar  1  -  tankl 

1 

yes 

tar  2  -  apcl 

1 

yes 

tar  3  -  m60 

1 

yes 

tar  4  -  van 

1 

yes 

31 

L1918 

260 

tar  1  -  tankl 

27 

yes 

tar  2  -  m60 

27 

yes 

32 

L19NS 

275 

tar  1  -  m60 

1 

yes 

tar  2  -  mantruck 

87 

yes 

tar  3  -  tankl 

147 

yes 

33 

L20  04 

368 

tar  1  -  apcl 

1 

yes 

34 

L20  08 

348 

tar  1  -  apcl 

1 

yes 

35 

L2017 

308 

tar  1  -  tankl 

116 

yes 

36 

L20_18 

448 

tar  1  -  tankl 

1 

yes 

37 

L20  20 

420 

tar  1  -  tankl 

1 

yes 

tar  2  -  target 

150 

yes 

38 

L2104 

760 

tar  1  -  bradley 

1 

yes 

tar  2  -  tankl 

70 

yes 

39 

L2115 

738 

tar  1  -  bradley 

1 

yes 

40 

L2117 

360 

tar  1  -  apcl 

1 

yes 

41 

L22  06 

348 

tar  1  -  tankl 

1 

yes 

42 

L22-08 

380 

tar  1  -  apcl 

1 

yes 

43 

L2210 

398 

tar  1  -  target 

1 

yes 

44 

L2214 

350 

tar  1  -  m60 

1 

yes 

tar  2  -  target 

1 

yes 

L23_12 

368 

tar  1  -  apcl 

1 

yes 

M14_06 

380 

tar  1  -  bradley 

1 

yes 

47 

M14_07 

400 

tar  1  -  bradley 

1 

yes 

48 

M1410 

498 

tar  1-  tankl 

1 

yes 

32 


tar  2  -  mantruck 

1 

yes 

49 

M1413 

380 

tar  1  -  mantruck 

1 

yes 

50 

M1415 

528 

tar  1  -  mantruck 

1 

yes 

tar  2  -  target 

1 

yes 

Appendix  F:  Publications  and  Theses  Generated  from  This  Project 

F.l.  Refereed  Journal  Publications 

1.  A.  Dawoud,  M.  S.  Alam,  A.  Bal  and  C.  Loo,  “Decision  Fusion  Algorithm  for  Target  Tracking 
in  Infrared  Imagery,”  accepted  for  publication,  Journal  of  Optical  Engineering,  Vol.  44,  2005. 

2.  A.  Bal  and  M.  S.  Alam,  “Dynamic  Target  Tracking  Using  Fringe-adjusted  Joint  Transform 
Correlation  and  Template  Matching,”  accepted  for  publication,  Journal  of  Applied  Optics, 
Vol.  43,  2004. 

3.  H.  C.  Loo  and  M.  S.  Alam,  “Invariant  Object  Tracking  Using  Fringe-adjusted  Joint 
Transform  Correlation,”  accepted  for  publication,  Journal  of  Optical  Engineering,  Vol.  43, 
2004. 

4.  M.  S.  Alam,  M.  Haque,  J.  Khan  and  H.  Kettani,  “Target  Tracking  in  Forward  Looking 
Infrared  (FLIR)  Imagery  Using  Fringe-adjusted  Joint  Transform  Correlation,”  Journal  of 
Optical  Engineering,  Vol.  43,  p.  1407-1413,  2004. 

5.  M.  S.  Alam,  J.  Khan  and  A.  Bal,  “Hetero-associative  multiple  target  tracking  using  fringe- 
adjusted  joint  transform  correlation,”  Journal  of  Applied  Optics,  Vol.  43,  p.  358-365,  2004. 

6.  Five  additional  journal  papers  are  currently  undergoing  review. 


F.2.  Conference  Publications  (Includes  1  Invited  Papers) 

1.  M.  S.  Alam  and  A.  Bal,  “Automatic  Target  Tracking  Using  Global  Motion  Compensation  and 
Fringe-adjusted  Joint  Transform  Correlator,”  to  appear,  Proceedings  of  the  SPIE  Conference 
on  Algorithms  and  Systems  for  Optical  Information  Processing  (part  of  49th  Annual  Meeting 
of  the  International  Society  for  Optical  Engineering),  Denver,  Colorado,  August  2004. 

2.  A.  Bal  and  M.  S.  Alam,  “Feature  extraction  technique  based  on  Hopfield  neural  network  and 
joint  transform  correlation,”  to  appear,  Proceedings  of  the  SPIE  Conference  on  Algorithms 
and  Systems  for  Optical  Information  Processing  (part  of  49th  Annual  Meeting  of  the 
International  Society  for  Optical  Engineering),  Denver,  Colorado,  August  2004. 

3.  A.  Bal  and  M.  S.  Alam,  “Automatic  Target  Tracking  in  FLIR  Image  Sequences,”  to  appear, 
Proceedings  of  the  SPIE  Conference  on  Automatic  Target  Recognition,  Orlando,  Florida, 
April  2004. 

4.  M.  S.  Alam,  E.  Horache  and  S.  F.  Goh,  “Performance  Evaluation  for  Cluttered  Infrared 
Images  Using  Fringe-adjusted  Joint  Transform  Correlation,”  Proceedings  of  the  SPIE 
Conference  on  Optical  Pattern  Recognition,  (part  of  2004  SPIE  Defense/Security 
Symposium),  Vol.  5437,  pp.  63-74,  Orlando,  Florida,  12-16  April,  2004. 


33 
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associative  multiple  target  tracking,”  to  appear,  Proceedings  of  the  SPIE  Conference  on 
Automatic  Target  Recognition,  Orlando,  Florida,  April,  2004. 

6.  H.  C.  Loo  and  M.  S.  Alam,  “Fringe-Adjusted  Joint  Transform  Correlation  Based  Invariant 
Target  Tracking  in  FLIR  Image  Sequences  (Invited  Paper),”  Proceedings  of  the  SPIE 
Conference  on  Optical  Pattern  Recognition  (part  of  2003  SPIE  Defense/Security 
Symposium),  Vol.  5437,  pp.  38-50,  Orlando,  Florida,  12-16  April,  2004. 

7.  A.  Dawoud,  M.  S.  Alam,  A.  Bal  and  C.  H.  Loo,  “Data  Fusion  Framework  for  Target  Tracking 
in  Airborne  Forward  Looking  Infrared  Imagery,”  to  appear,  Proceedings  of  the  SPIE 
Conference  on  Multisensor  Data  Fusion,  Orlando,  Florida,  April  2004. 

8.  M.  S.  Alam,  A.  El-saba,  E.  Horache  and  S.  Regula,  “Joint  Transform  Correlation  for 
Fingerprint  Identification  (Invited  Paper),”  to  appear,  Proceedings  of  the  SPIE  Conference  on 
Optical  Pattern  Recognition,  San  Jose,  California,  January  26-29,  2004. 

9.  M.  S.  Alam,  H.  Kettani,  M.  Haque,  J.  Khan,  A.  A.  S.  Awwal  and  K.  M.  Iftekharuddin, 
“Fringe-adjusted  JTC  Based  Target  Detection  and  Tracking  Using  Subframes  from  a  Video 
Sequence,”  Proceedings  of  the  SPIE  Conference  on  Photonic  Devices  and  Algorithms  for 
Computing  V  (part  of  48th  Annual  Meeting  of  the  International  Society  for  Optical 
Engineering),  Vol.  5201,  p.  85-96,  SPIE,  San  Diego,  California,  3-8  August,  2003. 

10. M.  S.  Alam,  “Parallel  optoelectronic  pattern  recognition  architectures  and  algorithms  (Invited 
Paper),”  Proceedings  of  the  25th  Conference  on  Opto-electro-techniques  and  Laser 
Applications,  p.  C01-C06,  IEEE,  Jakarta,  Indonesia,  October  2-3,  2002. 

11. M.  S.  Alam,  “Real  Time  Pattern  Recognition  and  Tracking  (Invited  Paper),”  Abstracts  of  the 
2002  Annual  Meeting  of  the  Optical  Society  of  America,  p.  122,  OS  A,  Orlando,  September 
29  -  October  4,  2002. 

12.  M.  S.  Alam,  “Class-associative  pattern  recognition  using  joint  transform  correlation  (Invited 
Paper),”  Proceedings  of  the  SPIE  Conference  on  Photorefractive  Fiber  and  Crystal  Devices: 
Materials,  Optical  Properties,  and  Applications  VIII  (part  of  47th  Annual  Meeting  of  the 
International  Society  for  Optical  Engineering),  p.  79-89,  Vol.  4803,  Seattle,  Washington,  7  - 
11  July,  2002. 
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3.  N.  Radhakrisna,  Thesis  Title:  Performance  Metrics  for  Pattern  Recognition  Algorithms, 
University  of  South  Alabama,  Graduation  Date:  August  2004. 

4.  J.  Khan,  Thesis  Title:  Class-associate  Fringe-adjusted  JTC  Based  Multiple  Target  Detection 
and  Tracking,  University  of  South  Alabama,  Graduation  Date:  May  2004. 
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Executive  Summary 

The  problems  of  automatic  target  detection,  tracking,  recognition  and  classification  have 
been  active  research  fields  in  image  processing  and  Neural  Networks  (NN).  In  this  report,  we 
explore  novel  ATR  techniques  such  as  object  pre-processing,  detection,  tracking  and 
classification  of  sequence  of  Infrared  (IR)  images  provided  by  the  Army  Missile  Command 
(AMCOM),  Huntsville  AL.  The  target  images  in  the  IR  database  contain  various  distortions  such 
as  translation,  scale  or  rotation  (both  in-plane  and  out-of-plane)  as  well  as  extensive  background 
clutter  and  noise.  We  present  image  pre-processing,  detection  and  tracking  algorithms  for  both 
single  and  multiple  objects  that  segment,  detect  and  track  the  object(s)  based  on  four  different 
techniques  such  as  i)  intensity;  ii)  correlation  time;  iii)  correlation  frequency;  and  iv)  Bayesian. 
For  the  first  three  techniques,  we  attempt  to  detect  and  track  the  targets  automatically  without  the 
prior  knowledge  of  the  location  and  size  of  the  object  in  the  reference  frame.  For  our  fourth 
technique  (Bayesian),  we  assume  full  knowledge  of  the  object  using  ground  truth  (GT)  data.  In 
addition,  we  exploit  an  elegant  mathematical  approach  using  Hilbert  transform  pair  of  wavelet 
bases  in  the  determination  of  exact  angle  of  rotation  for  targets.  We  compare  the  relative 
performance  of  Hilbert  wavelet,  Hilbert  transform  in  determination  of  in-plane  and  out-  of  plane 
rotation  angles.  Note  that  this  Hilbert-wavelet  analysis  is  performed  as  part  of  our  other  ongoing 
research  in  the  relevant  areas. 

In  our  proposed  detection  and  tracking  algorithms,  we  preprocess  the  image  sequences  to 
segment  the  object  from  the  background  clutter  and  noise  and  extract  relevant  statistical  and 
intensity  object  features.  We  enhance  our  algorithm  to  track  multiple  objects  in  image  frames 
based  on  the  knowledge  of  the  histogram  of  the  targets.  In  some  example  image  frames,  we 
obtain  detection  and  tracking  of  the  background  instead  of  the  objects  due  to  presence  of 
excessive  amount  of  background  clutter.  This  problem  of  background  detection  and  tracking  is 
magnified  for  the  first  three  techniques  (such  as  intensity,  correlation  time  and  correlation 
frequency)  wherein  the  initial  selection  of  the  reference  frame  is  quite  arbitrary.  However, 
improved  detection  and  tracking  performance  may  be  achieved  in  the  Bayesian  technique  since 
the  knowledge  of  the  target  location  in  the  first  frame  is  initially  known  from  the  GT  data.  We 
obtain  extensive  comparative  performance  metrics  of  our  proposed  techniques  in  detecting  and 
tracking  the  objects.  We  also  devise  graphical  user  interfaces  (GUI)  to  facilitate  such  comparison 
of  the  metrics. 

Finally,  we  take  advantage  of  our  detection  and  tracking  algorithms  to  extract  the  best 
possible  features  of  the  image  to  feed  to  the  neural  network  (NN)  for  classification.  In  this  work, 
we  use  self-organizing  map  (SOM)  and  k-nearest  neighbor  NN  techniques  to  classify  the  dataset. 
We  use  different  image  statistical  features  as  well  as  intensity  and  shape  of  the  object  for 
classification.  An  edge-tracing  method  is  also  investigated  which  sequentially  traces  all  the 
connected  points  in  the  edge  to  form  the  shape  of  the  image.  Our  classification  algorithms 
successfully  clusters  the  image  sequences  into  pre-defmed  classes  based  on  the  size  of  the  object. 
Note  that  the  success  of  any  NN  classification  depends  on  the  availability  of  representative 
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references  data  sets.  The  AMCOM  IR  datasets  do  not  contain  such  pristine  reference  image  of 
the  targets.  Thus,  the  goal  of  our  image  classification  in  this  project  has  been  to  identify  useful 
classification  methods  for  future  investigation. 

1.  Introduction 

The  problems  of  automatic  target  recognition  (ATR)  and  image  classification  have  been  active 
research  fields  in  image  processing  and  neural  networks.  The  image  classification  is  often  limited 
by  the  presence  of  background  clutter  noise.  Further,  the  image  may  be  subjected  to  different 
type  of  distortions  such  as  scale,  translation  and  rotation.  Thus,  the  researchers  of  animal  and 
machine  vision  have  researched  distortion-invariant  ATR  and  Image  Understanding  (IU) 
actively.  The  present  state  of  the  art  machine  vision  systems  do  not  even  approach  the 
performance  of  human  vision  in  IU,  which  implies  that  there  is  still  much  to  be  learned  from 
biological  vision  systems.  With  this  in  mind,  researchers  have  chosen  biomorphic-engineering 
approaches  such  as  Neural  Networks  to  solve  rather  intricate  ATR  and  IU  Problems.  ATR  is  the 
processing  and  understanding  of  an  image  in  order  to  recognize  the  targets.  The  long  time 
objective  of  ATR  research  is  to  derive  useful  approximations  or  bounds  on  the  performance  of 
ATR  systems  in  distortion-related  cases.  It  is  understood  that  the  most  accurate  estimates  of  the 
performance  of  a  particular  ATR  system  are  obtained  by  compiling  the  results  of  end-to-end 
systematic  experiments  on  rich  sets  of  measured  data.  Unfortunately,  the  end-to-end  experiments 
are  fiscally  and  computationally  expensive.  More  importantly,  the  task  of  measuring  or 
simulating  a  set  of  targets  over  the  variety  and  range  of  extended  operating  conditions  is  beyond 
current  capability.  Therefore,  almost  all  the  ATR  studies  involve  either  simulated  or  limited  scale 
ground  data  such  as  grayscale,  synthetic  aperture  radar  (SAR)  or  infrared  images. 

ATR  functionally  relies  in  some  way  on  its  knowledge  of  how  the  objects  and  classes  of 
interest  appear.  The  ‘comparison’  of  this  knowledge  to  the  current  information  on  new  object 
determines  the  results  of  the  recognition  process.  In  order  to  train  and  verify  the  performance  of 
a  classifier,  it  is  necessary  to  have  truth  data  for  all  image  objects  used  during  training  and 
verification.  Exogenous  information  such  as  fields  of  view  and  horizontal,  vertical  and 
depression  angles  can  be  used  to  a  greater  advantage  along  with  the  primary  data  stream.  Motion 
tracking  is  also  an  important  task  in  computer  vision  especially  when  objects  are  subjected  to 
certain  viewing  transformations.  Even  though  the  feature  space  maximizes  the  similarity  of 
objects  in  the  same  class  and  minimizes  the  similarity  of  objects  in  different  classes,  the  general 
nature  of  ATR  imagery  makes  feature  based  classification  a  very  hard  problem.  A  variety  of 
classification  schemes  have  been  implemented  and  evaluated.  Structural  constraints  reduce  the 
number  of  independent  weights  by  using  a  locally  connected  structure  or  sharing  the  same 
weights  on  many  connections. 

2.  Detection  and  Tracking  Results 
2.1  Intensity-based  Algorithm 

a)  Single  object  case 

We  apply  our  preprocessing  and  detection  algorithm  on  one  L17  20  of  the  IR  public 
image  database  [25],  These  are  the  scaled  and  rotated  images  from  the  database.  Fig.l  shows 
some  of  the  results  of  detection  and  tracking  obtained.  Out  of  778  images  in  the  database,  a  total 
of  750  images  are  detected  and  tracked  correctly.  We  have  applied  our  detection  and  tracking 
algorithm  on  other  sequences  of  the  database  [25]  and  obtained  the  similar  results. 
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Fig.  1  Detection  and  tracking  of  objects  at  different  scales  and  angles 

b)  Multiple  object  case 


- 

1 

| 

, 

[ 

m 


(C)  (d) 


Fig.  2  Detection  and  Tracking  multi-  object  case 

The  plots  in  the  Fig.  2  correspond  to  the  detection  and  tracking  of  2  different  images  in  multiple 
frames.  The  plots  on  the  top  left  comer  in  Fig.  2(a),  (b),  (c)  and  (d)  show  the  histogram  of  the 
raw  image.  The  peak  in  the  histogram  corresponds  to  the  intensities  of  the  most  dominant  object 
in  the  frames.  The  plots  on  the  lower  left  in  all  Figures  correspond  to  the  detection  and  tracking 
using  the  raw  images  with  no  intensities  removed.  The  plot  on  the  top  right  corresponds  to  the 
histogram  of  the  image  with  the  intensities  of  the  most  dominant  object  removed  while  the  plots 
on  the  lower  right  correspond  to  the  detection  and  tracking  of  next  dominant  object  in  the  frame. 
We  have  experimented  with  algorithm  to  detect  and  track  the  multiple  objects  on  other  image 
sequences  in  the  database  and  obtained  the  similar  results. 
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c)  Selection  of  Initial  Reference  frame  (Intensity  Algorithm) 

The  first  reference  frame  is  selected  from  the  sequence  L17_20  as  shown  in  Fig.  3(a). 
Note  that  this  selection  is  quite  arbitrary.  We  only  need  a  reference  frame  that  helps  us  in 


Fig  .3  a)  Raw  Reference  frame  (L1720,  frame  number  350)  and  b)  Pre-processed  frame  (L17  20,  frame  number 

500) 

locating  the  high  intensity  region  in  the  sequence  for  detection  and  tracking  of  targets.  It  is 
important  to  note  that  while  our  algorithm  performance  results  shown  herein  are  obtained  using  a 
reference  frame,  however,  this  requirement  is  not  mandatory.  To  prove  this  hypothesis  that  we 
really  do  not  need  any  specific  reference  frame,  we  have  done  pre-processing  on  another  frame 
as  shown  in  Fig.  3(b).  We  show  the  resulting  detection  and  tracking  in  Fig.  4. 

The  dark  box  corresponds  to  the  box  created  using  the  tracking  algorithm  and  bright  box 
is  the  box  created  using  the  GT  data.  Figure  4  and  suggests  that  the  pre-processing  is  good 
enough  to  select  ‘any’  reference  from  a  sequence  for  successful  detection  and  tracking. 


Fig.  4  Verification  of  detection  and  tracking  based  on  intensity  with  ground  truth  data. 

2.2  Bayesian  Algorithm 
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Figure  5  shows  the  detection  and  tracking  results  obtained  using  Bayesian  algorithm.  From  the 
Figure  it  is  evident  that  the  relative  change  in  intensity  is  not  much  and  hence  we  obtain  better 
detection  and  tracking  results.  Figure  6  shows  the  error  in  detection  and  tracking  of  the  object 
due  to  the  drastic  changes  in  the  intensity  using  algorithm. 


Fig.  6  Drastic  change  in  intensity  in  successive  frames  results  in  error  in  the  successive  frames  (Sequence  L22  08) 


Figure  7  shows  the  multi-target  detection  and  tracking  results.  From  the  Figure,  it  is  evident 
that  satisfactory  detection  and  tracking  results  have  been  obtained. 


Fig.7  Multiple  Target  Detection  and  Tracking  results  using  Bayesian  algorithm  (Sequence  LI 816) 


3.3  Hilbert-wavelet  based  Analysis 


a)  Out-of-plane  rotation  angle  detection 

We  first  select  the  278  image  frames  that  represent  out-of-plane  rotation  from  the  L17_20 
sequence  of  778  frames  at  regular  intervals.  Frames  in  Fig.  8  clearly  show  an  example  of  out-of- 
plane  rotation  and  scaling  since  the  image  in  the  first  frame  is  moving  towards  the  northeast 
direction  while  the  last  frame  is  moving  towards  northwest. 


Fig.  8  Out  of  plane  rotated  and  scaled  image  database 


From  the  results  obtained  in  Fig.9  (a),  we  find  that  there  are  only  two  cases,  wherein  the 
rotation  angles  have  been  improperly  determined.  A  close  observation  of  the  test  images  in  the 
database  suggests  that  objects  in  the  same  two  frames  do  not  have  enough  information.  Graphs 
with  corresponding  correlation  values  are  shown  in  Fig.  9(b).  The  two  dips  at  frame  numbers  41 
and  154  correspond  to  the  two  hikes  corresponding  to  two  confused  cases  in  Fig.  9(a). 
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(a)  (b) 

Fig  .9  Rotation  angle  determinations  with  Hilbert  wavelet  (a)  Frame  number  vs.  Rotation  angle  (b)  Frame  number 

vs.  Correlation 


b)  Improved  Detection  and  Tracking  (Distorted  case  using  Hilbert- wavelet) 

The  next  task  involves  the  detection  and  tracking  of  the  object  based  on  the  correlation 
value.  Note  that  this  Hilbert-wavelet  analysis  is  performed  as  part  of  our  other  ongoing  research 
in  the  relevant  areas.  The  tracking  of  the  target  improves  further  when  we  use  the  reference 
images  exactly  similar  to  the  test  image  as  shown  in  Fig.  8.  The  tracking  results  using  the  Hilbert 
wavelet  correlation  is  shown  in  Fig.  10.  Comparing  Fig.  5  and  Fig.  10  it  is  clear  that  objects  may 
be  tracked  by  using  reference  images  similar  to  test  cases  but  different  in  scale,  angle  (in  plane 
and  out  of  plane)  or  combination  of  both  and  more  efficient  tracking  can  be  achieved  by  using 
the  images  exactly  similar  to  the  test  cases. 


Fig.  10  Tracking  of  objects,  improved  cases. 


In  summary,  four  general  purpose  algorithms  and  one  mathematical  analysis  for  image  preprocessing,  detection  and 
tracking  the  targets  have  been  proposed.  We  discuss  two  of  these  techniques  in  detail  since  the  other  techniques 
involve  similar  steps.  The  results  obtained  for  detection  and  tracking  have  been  presented.  We  have  experimented 
our  detection  and  tracking  algorithms  on  other  sequences  in  our  database  and  obtained  the  similar  results.  Detection 
and  tracking  of  the  multiple  objects  in  the  frame  has  also  been  successful.  Most  of  the  frames  in  the  image  sequence 

have  been  detected  and  tracked  successfully. 


3.  Classification 

a)  SOM  Classification  with  Intensities  as  feature  vector 

Consider  the  extracted  intensity  features  obtained  from  the  IR  image  database  [25]  as 
shown  in  the  Fig  11. 


Fig.  11  Features  of  the  object  with  background  removed. 
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Fig.  1 1  shows  the  resulting  intensity  features.  The  dataset  consists  of  images,  which  contains  out- 
of-plane  rotation  and  scale  variations.  From  Fig.  1 1,  it  is  obvious  that  the  images  in  the  first  row 
show  the  scale  change  and  the  images  in  the  bottom  row  show  both  angle  and  scale  changes.  We 
take  the  images  at  regular  intervals  of  30  frames  in  a  sequence  of  378  images  for  training  such 
that  the  images  are  sufficiently  different  from  each  other.  Euclidean  distance  is  a  good  measure 
of  similarity  between  the  images.  Thus,  we  calculate  the  Euclidean  distance  as  shown  in  Table  1 
to  ascertain  that  the  resulting  eight  training  cases  are  sufficiently  different  from  each  other.  From 
the  Table  1  it  is  clear  that  only  training  cases  2  and  3  are  similar,  and  the  rest  of  the  cases  are 
sufficiently  different  from  each  other.  Thus,  ignore  training  case  3  and  trained  the  network  using 
the  rest  of  the  cases. 


Table  1.  Euclidean  distance  between  the  training  cases. 


Training 

Image 

1 

2 

3 

4 

5 

6 

7 

8 

1 

0 

0.0290 

0.0290 

0.0309 

0.0271 

0.0297 

0.0584 

0.0666 

2 

0.0290 

0 

2.4734e-004 

0.0027 

0.0133 

0.0206 

0.0337 

0.0486 

3 

0.0290 

2.4734e-004 

0 

0.0026 

0.0132 

0.0206 

0.0338 

0.0486 

4 

0.0309 

0.0027 

0.0026 

0 

0.0125 

0.0198 

0.0314 

0.0464 

5 

0.0271 

0.0133 

0.0132 

0.0125 

0 

0.0086 

0.0323 

0.0417 

6 

0.0297 

0.0206 

0.0206 

0.0198 

0.0086 

0 

0.0321 

0.0375 

7 

0.0584 

0.0337 

0.0338 

0.0314 

0.0323 

0.0321 

0 

0.0214 

8 

0.0666 

0.0486 

0.0486 

0.0464 

0.0417 

0.0375 

0.0214 

0 

Examples  of  3  different  clusters  out  of  8  are  shown  in  Fig.  12  with  cluster  centers  along 
with  corresponding  few  images  in  each  cluster.  From  the  classification  plots  in  Fig.  12,  it  is 
obvious  that  the  similar  images  have  been  clustered  in  one  cluster.  Images  in  cluster  1  are  the 
ones  after  the  image  has  undergone  out-of-plane  rotation  of  90°  while  moving  along  y-axis. 
Images  at  smaller  scales  are  grouped  in  cluster  2,  while  those  at  an  intermediate  scale  are 
grouped  in  cluster  3.  The  images  in  cluster  7  are  at  a  higher  scale  and  those  in  cluster  8  are  at 
smaller  scales  and  are  moving  at  0  degrees  with  respect  to  y-axis  respectively.  The  images  in 
cluster  4  are  the  ones  when  the  image  was  undergoing  some  angle  change  with  respect  to  y-axis. 
Cluster  5  captured  the  occluded  objects.  They  are  classified  as  the  images  which  have  undergone 
out-of-plane  rotation  of  90°  but  smaller  in  scale.  Finally,  cluster  6  consists  of  predominantly  the 
noise  points.  _ 


Fig.  12  Cluster  plots  for  Image  sequence  database  1. 
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b)  SOM  Classification  with  Shape  as  feature  vector 

As  discussed  earlier  we  keep  the  first  30  DCT  coefficients  to  retain  most  of  the  information.  The 
training  set  is  selected  manually  and  care  has  been  taken  that  they  are  taken  at  regular  intervals 
of  30  so  that  they  can  be  sufficiently  different  from  one  another.  We  determine  the  Euclidean 
distance  among  the  images  in  order  to  find  how  similar  or  dissimilar  the  images  are  as  shown  in 
Table  2. 


Table  2.  Euclidean  distance  between  training  cases 


Training 

image 

1 

2 

3 

4 

5 

6 

7 

8 

1 

0 

5.1710 

2.7832 

6.0188 

6.6933 

7.5902 

38.3086 

69.9684 

2 

5.1710 

0 

3.6456 

1.7805 

1.9442 

3.3205 

33.5627 

66.8063 

3 

2.7832 

3.6456 

0 

4.7818 

5.0050 

5.9175 

36.4828 

69.5037 

4 

6.0188 

1.7805 

4.7818 

0 

2.0195 

3.3377 

32.9519 

65.9773 

5 

6.6933 

1.9442 

5.0050 

2.0195 

0 

2.1245 

32.0379 

66.3667 

6 

7.5902 

3.3205 

5.9175 

3.3377 

2.1245 

0 

31.1081 

67.1602 

7 

38.3086 

33.5627 

36.4828 

32.9519 

32.0379 

31.1081 

0 

58.3008 

8 

69.9684 

66.8063 

69.5037 

65.9773 

66.3667 

67.1602 

58.3008 

0 

From  the  Table  2,  it  is  evident  that  the  training  images  are  dissimilar  enough  to  be 
considered  as  different  training  cases.  When  we  train  the  SOM  using  the  different  training  cases, 
the  similar  shapes  are  grouped  as  centers  of  nearby  clusters  while  the  dissimilar  shapes  are 
grouped  in  different  clusters.  From  Fig.  13  (2  out  of  six  are  shown),  it  is  clear  that  the  images 
belonging  to  the  similar  shape  are  classified  as  one  cluster.  The  images  in  cluster  1  show  the 
shape  of  the  image  after  the  image  has  undergone  90°  rotation  with  respect  to  its  original  motion. 


Cluster  2  consists  of  images  that  may  not  be  traced  properly.  It  also  includes  the  images  that  are 
similar  to  mis-traced  images.  Cluster  4  consists  of  images  that  are  partially  occluded.  Cluster  3, 
5,  6  have  images  of  similar  shape  but  cluster  5  has  mis-traced  images.  From  our  observation  of 
the  dataset  we  find  that  the  object  moves  at  an  angle  of  90°  with  respect  to  its  original  motion 
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Traced  edge 


Fig.  14(a)  Edge  and  (b)  Mis-traced  edge. 

after  frame  number  300  (out  of  375  frames)  are  the  ones  after  the  image  has  moved  at  an  angle  of 
90°  with  respect  to  its  original  motion.  When  we  plot  the  traced  contour  of  these  confused  cases, 
we  observe  that  image  frames  are  not  traced  correctly  as  shown  in  Fig.  14.  As  a  result,  these 
partially  edge-traced  frames  were  grouped  in  the  wrong  clusters  by  SOM  classifier. 

c)  k-NN  Classification  with  Shape  as  feature  vector 

The  results  of  the  k-nearest  neighbor  classifier  are  obtained  and  results  obtained  by  varying  the 
usage  of  statistical  features  are  observed. 

i)  Results  using  intensity  features 

We  show  an  example  classification  result  using  the  intensity  features  form  L1720  data 
sequence  below. 


Fig.  15  Class  vs.  image  number  using  all  intensity  Fig.  16  Classes  vs.  image  number  using  statistical 

features  features 

Figure  15  shows  how  many  images  are  classified  under  in  each  of  the  8  classes  that  we  have  pre¬ 
defined.  The  description  of  the  individual  classes  is  shown  in  Fig.  16. 
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Fig.  17  Example  of  different  classes 


Cluster  1  consists  of  images  with  improper  segmentation.  Cluster  2  consists  of  images  when  the 
object  has  not  undergone  rotation  at  higher  scale.  Cluster5  shows  images  when  object  is  just 
making  a  turn.  Cluster  8  shows  objects  at  high  scale  when  object  has  undergone  90  degrees 
rotation  and  it  also  shows  occluded  objects  classified  correctly  as  one  of  non  occluded  objects. 

ii)  Results  using  statistical  features 


We  show  an  example  classification  result  using  the  statistical  features  such  as  mean,  variance, 
maximum,  minimum  form  L17_20  data  sequence  in  Fig.  17.  All  the  intensity  features  are  now 
replaced  by  statistical  features.  The  image  (region  of  interest  (57  x  57))  is  now  taken  and  divided 
into  blocks  of  8  x  8  and  each  block  is  replaced  by  statistical  features.  Each  of  these  statistical 
features  are  first  normalized  by  dividing  the  whole  matrix  by  maximum  number  in  that  matrix. 
This  is  done  in  order  that  one  particular  feature  not  have  major  impact  on  classification.  The 
results  obtained  are  similar  to  the  ones  shown  above. 

The  improperly  segmented  images  now  fall  into  cluster2.  Some  of  the  images  from  cluster 
1  (previously  using  all  intensity  features)  are  now  classified  under  cluster  3.  Careful  observation 
of  data  shows  that  there  is  not  much  change  between  images  of  cluster  1  and  cluster  3 
respectively  as  shown  in  Fig.  18. 


4.  Performance  Metrics 

We  obtain  the  Receiver  Operating  Curves  (ROC)  metrics  for  our  Intensity-based 
algorithm  as  shown  below. 


4.1  Results  for  the  ROC  curves 

We  plot  detection  and  tracking  performance  vs.  filter  size  and  filter  resolution  in  Figs.  20 
and  2 1  respectively. 
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Fig  19  Performance  of  1)  Minimum  filter,  2) 

Median  filter,  3)  Max  of  east  west  north  south  filter  resolutions 

and  4)  Maximum  filter  for  varying  window  size. 


Fig  20  Performance  of  the  filters  at  different 


In  Fig.  19  note  that  the  x-axis  variable  represents  the  following  format  (3  represents  3  x  3,  4  represents  4x4  and  so 
on)  for  the  size  of  the  window.  From  the  plot  is  clear  that  the  maximum  filter  for  the  window  size  of  3  x  3  offers  the 
best  tracking  performance.  We  perform  the  detection  and  tracking  algorithm  for  various  resolutions  of  the  images. 
The  resolution  increases  as  we  move  from  left  to  right  along  the  x-axis.  From  the  Fig.  20,  it  is  evident  that  increasing 
the  resolution  offers  better  tracking  for  all  types  of  rank-order  filters  considered.  From  Fig.  21  it  is  clear  that  higher 
percent  of  compression  yields  increasing  percent  of  mis-tracking. 


for  varying  compression 


Fig.  22  Signal  to  noise  ratio  of  the 


images  taken  at  regular 
intervals  of  30 


Fig.  23  Performance  of  the  classifier 


for  varying  window  size 


The  SNR  is  obtained  and  it  is  plotted  in  Fig.  22  for  images  at  regular  intervals  of  30  for  a 
sequence  of  778  images.  As  the  potential  target  size  increases  in  the  frame  sequence  (change  of 
scale),  the  signal  to  clutter  ratio  also  increases  as  shown  in  Fig.  23  as  the  target  becomes  larger. 
From  Fig.  23  it  is  clear  that  smaller  block  size  offers  better  classification. 


4.4  Composite  Metrics 

We  summarize  our  results  and  experiments  of  our  proposed  algorithm  in  Table  3.  Table  3 
shows  the  summary  of  most  of  the  results  obtained  in  this  work.  From  the  table  it  is  obvious  that 
discarding  about  50  %  of  the  coefficients  offers  a  reasonable  result.  The  classification  using  the 
SOM  gave  a  reasonable  classification  of  about  89  %  using  statistical  features  and  about  82.16  % 
using  the  shape  features.  The  rest  of  the  results  are  obvious. 
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Table  3.  Summary  of  results. 


Image  sequence 

In-plane  rotation  identification 
results 

Out-of  plane  rotation  identification 

Tracking 

Classification 

SOC 

MOC 

Seg. 

T*  s/f 

% 

Avg. 

%of 

T*  s/f 

%  M- 

Avg. 

%  M-Tr. 

Features 

SOM 

M- 

corr 

Comp. 

det 

corr 

(%  of 

det 

classification) 

L1720 

- 

No 

2.881 

0 

91.6 

0 

2.00 

0.72 

99.2 

0.64 

Intensity 

89.92 

Shape 

82.16 

L1720 

- 

No 

2.822 

0 

90.8 

50 

1.72 

0.72 

99 

- 

- 

- 

LI  720 

- 

No 

2.08 

0 

82.3 

80 

1.117 

47.1 

90 

- 

- 

- 

LI  720 

- 

No 

1.999 

0 

77.8 

90 

1.01 

87.98 

87.87 

- 

- 

- 

L1720 

- 

Yes 

0.460 

0 

74.6 

0 

1.516 

67.27 

95.3 

- 

- 

- 

- 

T.1R1R 

No 

.  2.880 

0 

95.5 

0 

2.01 

0.38 

92.38 

6 

- 

- 

- 

L 1 8 1 8 

Yes 

2.822 

0 

94.3 

50 

1.72 

0.38 

90.36 

- 

- 

- 

- 

LI  8 1 8 

Yes 

2.08 

0 

89.1 

80 

1.117 

37.33 

90.28 

- 

- 

- 

- 

LI  8 1 8 

Yes 

1.999 

0 

88.02 

90 

1.01 

79.97 

87.82 

- 

- 

- 

*933  MHz  p-3  xeon  processor  with  256  MB  of  memory 


SOC:  Single  object  case. 

MOC:  Multiple  object  case. 

Seg.  :  Segmentation. 

T  s/f:  Time  in  seconds  per  frame 
%  M-Tr:  Percentage  of  Mistracking 
(Ratio  of  number  of  images,  which 
have  not  been  tracked  to  the  total 
number  of  images). 


%  M-det:  Percentage  of  mis-detection  (Ratio  of  number 
of  images  whose  angle  has  been  identified  wrongly  to  the 
total  number  of  images). 

Avg.  corr:  Average  correlation. 

%  of  Comp:  Percentage  of  compression  (  Ratio  of  number 
of  discarded  coefficients  to  the  total  number  of 
coefficients). 

SOM:  Self  organizing  map. 

T  s/f:  Time  in  seconds  per  frame 


4.5  Comparison  Metrics 

In  Table  4  below,  we  summarize  the  comparative  performance  metrics  of  our  four 
general  purpose  detection  and  tracking  algorithms  described  above  for  all  50  data  sequences  in 
the  AMCOM  database. 


Results  obtained  at  U  of  M 


ID  No. 

Sequence 

name 

No.  of 
frames 

No.  of 
target 
s 

description 
of  targets 
in  the 
sequence 

SNR  (# 
of  target 
pixels 
divided 
by 

backgrou 

nd) 

%of 

Detection 

and 

Tracking 

(Intensity 

algorith 

m) 

%  of 
Detection 
and 
Tracking 
(correlati 
on_algori 
thm) 

%of 

Detection 

and 

Tracking 

(bayesian 

algorith 

m) 

%of 

detection 

and 

tracking 

corr_freq 

uency 

algorithm 

%of 

Detection 

and 

tracking 

(manual_c 

ombined_s 

ummary) 

%of 
Detection 
and  tracking 
(automated 
_combined_ 
summary) 

1  L15_20  215  1  tank 

0.0127 

tank 

85.58% 

tank  83% 

tank 

100% 

tank 

18.13% 

tank  - 
100% 

tank  -  74% 

2  L15_NS  320  2  tankl,  0.0097 

truckl 

tankl 
76.42%  & 
truckl  - 
9% 

tankl- 
75.4% 
truckl - 
9.5% 

tankl  (75/ 
320)23%, 
truck(8/2 
0)  40% 

tankl- 

38.75% 

tankl - 
75%;  truck 
40% 

tankl  -  65%; 
truck  55% 

3  L16_04  400  1  tankl  x 

no  binary 
files 
exists 

66% 

89% 

30.57% 

no  GT  to 
compare 

no  GT  to 
compare 

4  L16_07  410  3  tankl,  0.0093 

mantruck, 

tankl 

tankl 

86% 

mantruck 

77.98% 

tankl 

30% 

tankl 

90% 

mantruck 

81.91% 

tankl 

30% 

tankl  (24 
4/410) 
60% 
mantruck 
(386/386) 
100% 

tankl - 
22.7% 

tankl  - 
90%; 
mantruck 
100%; 
tankl  - 
30% 

tankl  - 
86%; 
mantruck 
78%;  tankl 
-  30% 
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5 

L16_08 

290 

3 

m60,  apcl, 
mantruck 

0.0175 

mantruck 

67.8% 

m60 

100% 

apcl 

32% 

mantruck 

69.38% 

m60 

100% 

apcl 

31.33% 

mantruck 

(37/98)38 

% 

m60(290/ 

290)100 

%;apc1(5 

2/83)63% 

m60%- 

50% 

m60- 
100%; 
apcl  - 
84%; 

mantruck  - 
69% 

m60-100%; 
apcl  - 
100% 

6 

LI  6 _ 1 8 

300 

3 

apcl,  m60, 
truck 

0.0184 

apcl  for 
83.85%, 
m60  for 
12.12% 

apcl 
83%  m60 
13.33% 
truck  0% 

apcl  (291 
/291  )1 00 
%;m60(3 
4/102)33 
%truck(7/ 
7)100% 

apcl- 

40.54% 

apcl- 

100%; 

m60-33%; 

truck- 

100% 

apcl -100%; 
m60-95%; 
truck-1 00% 

7 

L 1 7 _ 0 1 

388 

2 

bradely,pic 

kup 

0.0125 

bradely 

84.7% 

pickup 

0% 

bradely 

95.14% 

pickup 

0% 

bradely(1 

00/371)2 

7% 

pickup(16 

/46)35% 

bradely- 

41.5% 

bradely- 

95%; 

pickup- 

42% 

bradely- 

75%; 

pickup-0% 

8 

L17_02 

698 

5 

Man 

pickup 

bradely 

0.0047 

mantruck 

53.9% 

and 

bradely 

27% 

mantruck 

1  40% 
pickup 
0% 
bradely 
30% 

mantruck 
(273/459) 
60%,  pick 
up(46/69) 
67%,  brad 
ely(21 4/3 
43)62%,  t 
ank(50/5 
0)100% 

mantruck 
-  33.2% 
tankl  - 
8.1% 
bradely- 
40.24% 

mantruck- 
60%; 
bradely- 
62%;  tank- 
100%; 
pickup- 
67% 

mantruck- 
59%; 
bradely- 
49%;  tank- 
100%; 
pickup-0% 

9 

L17_20 

778 

2 

target, 

m60 

0.011 

m60 
70.5% 
target  0% 

target 

5.7% 

m60 

71.16% 

target(0/3 

5)0% 

m60(743/ 

743)100 

% 

target- 

73.6% 

m60- 

69.45% 

target- 

73.6%; 

m60-100% 

target-1 4%; 
m60-78% 

10 

L18_03 

450 

1 

bradely 

0.0046 

bradely 

58% 

bradely 

69.12% 

27% 

bradely- 

54.8% 

bradely- 

69% 

bradely- 

64% 

11 

L18_05 

779 

4 

tankl, 
apcl,  m60, 
tank 

0.0053 

no  GT 
data 

no  GT 
data 

no  GT 
data 

no  GT 
data 

no  GT  to 
compare 

no  GT  to 
compare) 

12 

LI  8-07 

260 

1 

bradely 

0.00952 

bradely 

94% 

bradely 

97% 

bradely 

78% 

bradely- 

36.5% 

bradely- 

97% 

bradely- 

96% 

LI  8 _ 1 2 

300 

3 

bradely, 
tankl,  m60 

0.0108 

bradely 

96% 

bradely 

86.66% 

tankl 

15.08%; 

m60 

8.6% 

bradely(3 
00/300)1 
00%;tank 
1(202/20 
2)100%; 
m60(94/9 
4)  100% 

bradely- 

37.66% 

bradely- 

100%; 

tankl- 

100%; 

m60-100% 

bradely- 
97%;  tankl - 
92%;  m60- 
100% 

14 

LI  8_1 3 

326 

3 

tankl, 
m60,  apcl 

0.0204 

tankl 
82.52% 
m60  48% 
apcl 
22% 

apcl- 
26.58% 
tankl  - 
84.59% 
m60- 
53.04% 

apcl- 

(78/170) 

46% 

tankl  - 
40.97% 

tankl  - 
85%;  m60- 
53%; 

apcl -46% 

tankl -85%; 
m60-95%; 
apcl -83% 

15 

LI  8 _ 1 5 

seq-1 

339 

4 

m60,  truck, 
tankl, 
bradely 

0.0098 

tankl- 

68.9% 

bradely 

75% 

m60- 

45.62% 

truck-9% 

tankl- 

81.75% 

bradely- 

74.886% 

m60- 

(238/238) 

100% 

tankl- 

(103/285) 

36% 

m60- 

43.67% 

tankl- 

20.33% 

bradely- 

18.7% 

m60- 

100%; 

truck- 

100%; 

tankl- 

82%; 

bradely- 

100% 

m60-99%; 
truck-92%; 
tankl -97%; 
bradely- 
85% 

truck(12/ 
12)  100% 
bradely(1 
87/187) 
100% 

48 


16  LI  8 _ 1 5  342  3  m60,  0.0121 

seq-2  tankl, 

bradely 

tankl 

87% 

m60- 
46.63% 
tankl  - 
84.27% 
bradely- 
77.77% 

m60 

(243/243) 

100% 

tankl 

(208/314) 

66% 

bradely 

(181/191) 

95% 

m60- 
2.9% 
tankl - 
27.24% 
bradely- 
29.25% 

m60- 
100%; 
tankl - 
87%; 
bradely- 
95% 

m60-100%; 
tankl -90%; 
bradely- 
90% 

17  LI  8-1 7  240  2  tankl ,  m60  0.0145 

seq-1 

different 
objects 
for  66% 

tankl- 

57.9% 

m60-96% 

tank1(12 

0/240) 

50% 

m60(198/ 

198) 

100% 

tank- 

2.5% 

tankl- 
58%;  m60- 
100% 

tankl -88%; 
m60-100% 

18  LI  8 1 7  266  2  tank1,m60  0.00153 

seq-2 

no  gt.lis 
file 
various 
objects 
70% 

no  gt.lis 
file 

no  gt.lis 
file 

no  GT.Iis 
file 

no  GT  file 
to 

compare 

tankl- 
62%;  m60- 
100% 

no  GT  file  to 
compare 

19  LI  8 _ 1 6  328  2  tank1,m60  0.0205 

tankl- 
44.9% 
and  m60 
20.43% 

tankl- 

61.58% 

m60- 

51.54% 

tankl 

(145/329) 

44% 

m60 

(290/290) 

100% 

tankl- 

22.86% 

tankl -53%; 
m60-99% 

20  LI  8 1 8  365  6  mantruck,  0.0224 

target, 
apcl,  m60, 
tankl, 
testvan 

tracking 
algorithm 
confuses 
for  first 
140 
frames 
tankl  for 
78.1% 

mantruck 

-4.2% 

apcl- 

24% 

m60- 

21.77% 

tankl- 

81.86% 

testvan- 

0% 

target- 

25% 

mantruck 
-  (33/44) 
75%, 
target  - 
(3/4) 
75%, 
apcl  - 
(56/95) 
59%, 
m60  - 
(44/121) 
37%, 
tankl  - 
(205/213) 
96%, 
testvan  - 
(4/4) 
100% 

mantruck 
-  7.4% 
target- 
9.6% 
apcl- 
5.31% 
m60- 
1 1 .4% 
tankl- 
13.6% 

mantruck- 

75%; 

target- 

75%; 

apcl -59%; 
m60-37%; 
tankl - 
96%; 
testvan- 
100% 

mantruck- 
60%;  target- 
75%;  apcl- 
66%;  m60- 
53%;  tankl - 
99%; 
testvan- 
100% 

21  L 1 9_0 1  240  1  tankl  0.0145 

Tankl 

74.3% 

tankl- 

68.3% 

Tankl 

0% 

tankl- 

53.33% 

tankl -74% 

tankl -79% 

22  L19_02  270  1  tankl  0.0359 

tankl 

10% 

tankl- 

84.7% 

Tankl 

0% 

tankl- 

31.88% 

tankl -85% 

tankl -91% 

23  L19_04  270  1  tankl  0.0454 

tankl 

80.74% 

tankl- 

73.34% 

Tankl 

37% 

tankl - 
44.44% 

tankl -81% 

tankl -78% 

24  L19_06  265  4  tankl,  0.041 

apcl, 

mantruck, 

van 

tankl  for 
27.42% 

tankl - 
84.9% 
apcl- 
92.4% 
mantruck 
-65% 
van-0% 

tankl 

(265/265) 

100% 

apcl 

(238/265) 

90% 

mantruck 

(69/197) 

35% 

van 

(20/20) 

100% 

tankl - 
34.45% 

tankl - 
100%; 
apcl -90%; 
mantruck- 
65%;  van- 
100% 

tankl -69%; 
apcl -83%; 
mantruck- 
61%;  van- 
100% 

25  L19_07  195  1  mantruck  0.0441 

mantruck 

14.53% 

mantruck 

-7% 

mantruck 

94% 

mantruck 
-  58.8% 

mantruck- 

94% 

mantruck- 

83% 
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26  LI  9 _ 1 0  130  2  tankl,  0.0413 

apcl 

tankl 

87% 

apcl- 
51.35% 
tankl  - 
83.33% 

tankl 

(130/130) 

100% 

apcl 

(90/90) 

100% 

tankl- 

41.22% 

tankl- 

100%; 

apcl- 

100% 

tankl- 
100%; 
apcl -100% 

27  LI  9 _ 1 1  165  2  tankl,  0.0874 

apcl 

tankl  for 
93.94% 

tankl- 

95.15% 

apcl- 

98.78% 

tankl 

(165/165) 

100% 

apcl 

(165/165) 

100% 

tankl - 
23.6% 

tankl - 
100%; 
apcl- 
100% 

tankl - 
100%; 
apcl -100% 

28  LI  9 1 3  265  4  tankl,  0.0474 

apcl,  m60, 
mantruck 

tankl  for 
77.78%  t 

tankl- 

83.9% 

apcl- 

86.79% 

m60-80% 

mantruck 

-40% 

tankl 

(106/264) 

40% 

apcl 

(208/264) 

79%; 

m60 

(35/66) 

53% 

mantruck 

(12/12) 

100% 

tankl- 

32.07% 

tankl - 
84%; 

apcl -87%; 
m60-80%; 
mantruck- 
100% 

tankl -93%; 
apcl -84%; 
m60-82%; 
mantruck- 
100% 

29  LI  9 _ 1 5  350  5  tankl,  0.0454 

apcl,  m60, 
van, apcl 

tankl  for 
70.86% 

m60- 

63.41% 

tankl- 

63.6% 

van- 

59.44% 

apcl- 

54.16% 

m60 

(184/347) 

53% 

tankl 

(129/332) 

39% 

apcl 

(76/189) 

40% 

van 

(65/216) 

30% 

m60- 
26.7% 
apcl- 
25.2% 
tankl - 
20.4% 
van- 18% 

tankl- 

71%; 

apcl -54%; 
m60-63%; 
van-59%; 
apcl -54% 

tankl -86%; 
apcl -84%; 
m60-86%; 
van-74%; 
apc-72% 

30  LI 9 _ 1 8  260  2  tankl,  m60  0.0078 

tankl 
60%  m60 
98% 

tankl 
52.99% 
m60  90% 

tankl 

(54/260) 

21% 

m60 

(12/12) 

100% 

tankl - 
18.37% 

tankl  - 
60%;  m60- 
100% 

tankl  - 
48%;  m60- 
100% 

31  L19_NS  275  3  m60,  0.0509 

mantruck, 

tankl 

different 

objects 

78.6% 

m60- 
63.79% 
mantruck 
-62.6% 
tankl - 
77.51% 

m60 

(40/50) 

80% 

mantruck 

(117/137) 

85% 

tankl 

(41/99)41 

% 

m60- 

50.24% 

mantruck 

-37.13% 

tankl- 

32.8% 

m60-80%; 

mantruck- 

85%; 

tankl -78% 

m60-91%; 
mantruck- 
62%;  tankl - 
68% 

32  L20_04  368  1  apcl  0.0039 

apcl- 

87% 

apcl- 

83% 

apcl- 

100% 

apcl- 

26.35% 

apcl- 

100% 

apcl -66% 

33  L20_08  348  1  apcl  0.0121 

apcl 

60.81% 

apcl- 

44.8% 

no  object 

apcl- 

11.49% 

apcl -61% 

apcl -32% 

34  L20_1 7  308  1  tankl  0.0108 

tankl  - 
100% 

tankl- 

96.3% 

no  object 

tankl- 

23.43% 

tankl - 
100% 

tankl -66% 

35  L20_1 8  448  1  tankl  0.0034 

tankl- 

100% 

tankl- 

85% 

72% 

tankl- 

25.66% 

tankl- 

100% 

tankl -59% 

36  L20_20  420  2  tankl,  0.0037 

target 

target- 

72% 

no  gt 
data 

no  gt 
data 

no  GT 
data 

no  GT  to 
compare 

no  GT  to 
compare 
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37 

L21_04 

760 

2 

bradely, 

tankl 

0.0087 

follows 
the  gt 
data 
100% 

bradely- 
17.55% 
tankl - 
88.5% 

bradely 

(44/321) 

14% 

tankl 

(481/746) 

65% 

bradely- 
1.4% 
tankl -0% 

bradely- 

18%; 

tankl -89% 

bradely- 
35%;  tankl - 
89% 

38 

L2 1 _  5 

738 

1 

bradely 

0.0105 

bradely- 

98.8% 

bradely- 

93.76% 

hard  to 

see 

object 

bradely- 

21% 

bradely- 

98% 

bradely- 

89% 

39 

L21 _  7 

360 

1 

apcl 

0.0087 

apcl- 

96.11% 

apcl- 

94.44% 

apcl- 

100% 

apcl- 

47.77% 

apcl- 

100% 

apcl -74% 

40 

L22_06 

348 

1 

tankl 

0.0237 

tankl  - 
47.42% 

tankl - 
17% 

tankl - 
77% 

tankl - 
45.4% 

tankl -77% 

tankl -36% 

41 

L22_08 

380 

1 

apcl 

0.0219 

apcl- 

98.1% 

apcl- 

93% 

apcl  - 
100% 

apcl- 

67.1% 

apcl- 

100% 

apcl -100% 

42 

L22_1 0 

398 

X 

no  Gt 
data 

no  Gt 
data 

no  Gt 
data 

no  GT 
data 

no  GT 
data 

no  GT  data 

43 

L22_14 

350 

2 

m60, 

target 

0.0221 

m60- 

54.18% 

target- 

43% 

target  - 
26.31% 
m60- 
48.175% 

m60 

(112/273) 

41% 

target 

(75/75) 

100% 

m60- 

12.68% 

target- 

14.86% 

m60-54%; 

target- 

100% 

m60-42%; 
target-1 00% 

44 

L23_1 2 

368 

1 

apcl 

0.0071 

apcl- 

82.1% 

apcl- 

89.4% 

apcl- 

100% 

apcl- 

82.88% 

apcl- 

100% 

apcl -87% 

45 

M 1 4_06 

380 

1 

bradely 

0.0121 

bradely- 

53.23% 

bradely- 

61.49% 

bradley- 

100% 

bradely- 

36% 

bradely- 

100% 

bradely- 

100% 

46 

M 1 4_07 

400 

1 

bradely 

0.0125 

bradely- 

66.84% 

bradely- 

57.43% 

breadely 

100% 

bradely- 

21.42% 

breadely- 

100% 

breadely- 

100% 

47 

M 1 4_1 0 

498 

2 

tankl, 

mantruck 

0.0236 

tankl- 

68.1% 

tankl- 

73.8% 

mantruck 

-57.14% 

tankl- 

40% 

mantruck 

-50% 

tankl- 

66.93% 

tankl- 

74%; 

mantruck- 

57% 

tankl- 

100%; 

mantruck- 

99% 

48 

M14_13 

380 

1 

mantruck 

0.0147 

mantruck 

-51.89% 

mantruck 

-76.34% 

mantruck 

-30% 

mantruck 
-  27.36% 

mantruck- 

76% 

mantruck- 

76% 

49 

LI  4 _ 1 5 

281 

1 

mantruck 

0.0127 

mantruck 

-76.16% 

mantruck 

-83.6% 

mantruck 

-100% 

mantruck 
-  52.44% 

mantruck- 

76% 

mantruck- 

76% 

50 

M 1 4_1 5 

528 

2 

mantruck, 

target 

0.0132 

target- 

38% 

mantruck 

-56.37% 

target-0% 

mantruck 

(35/380) 

9% 

target 

(20/20) 

100% 

mantruck 
-  45.24% 

mantruck- 

56%; 

target- 

100% 

mantruck- 
53%;  target- 
84% 

5.  Conclusion 

In  this  work,  we  first  propose  an  image  pre-processing  and  segmentation  algorithm  to  remove 
most  of  the  background  noise  and  clutter  from  IR  image  dataset.  We  then  introduce  four  novel 
algorithms  and  an  elegant  mathematical  technique  to  detect  and  track  the  objects  of  interest.  We 
also  introduce  techniques  for  building  the  tracking  box  to  capture  the  image  of  interest  and 
extraction  of  the  features  for  classification  once  the  target  is  detected.  The  proposed  algorithms 
have  been  used  on  different  image  sequences  in  the  IR  database,  and  the  resulting  image 
detection  and  tracking  performance  has  been  summarized  extensively.  Our  single  object-based 
algorithm  exploits  the  image  correlation  statistics  for  tracking  the  most  dominant  object  in  the 
frame.  Based  on  the  knowledge  of  the  histogram,  we  have  extended  our  algorithm  to  track 
multiple  objects  in  a  frame.  In  the  unlikely  but  possible  event  when  two  objects  contain  same 
intensity,  our  tracking  algorithm  may  track  one  object  or  the  other  object  at  a  time.  Further,  if  the 
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similar  intensities  are  present  in  the  background  as  well  as  in  dominant  object,  intensities  of 
object  may  be  lost  due  to  the  pre-processing,  and,  hence  it  is  never  tracked.  For  an  example,  for 
LI 81 6,  even  though  we  observe  only  one  window  for  tracking,  it  corresponds  to  more  than  one 
object,  and  the  random  placement  of  the  window  is  due  to  the  existence  of  highest  frequency 
component  randomly  occurring  in  the  frame.  Thus,  it  appears  that  the  same  window  is  tracking 
different  objects. 

As  mentioned  above,  we  do  not  assume  any  prior  knowledge  of  the  target  in  our  Intesity, 
Correlation_time  and  Correlation_frequency  algorithms  while  we  assume  full  knowledge  of 
target  location  in  our  Bayesian  algorithms  using  the  GT  data.  Off  course,  we  pay  a  price  for  this 
uncertainty  of  knowledge  of  the  target  in  our  first  three  algorithms-  i.e.,  we  obtain  relatively  poor 
performance  as  evidenced  in  the  performance  metrics  summary  results.  Thus,  we  detect  and 
track  background  sometimes  in  the  first  three  algorithms.  Note  that  in  Intensity  algorithm  we 
detect  and  track  one  target  at  a  time  while  in  Correlation_time  algorithm,  we  detect  and  track  two 
dominant  intensity  targets  at  a  time.  At  this  time,  this  appears  to  be  a  reasonable  limit  of  our 
image  preprocessing  and  segmentation  function  that  retains  only  enough  intensity  information  of 
two  targets  at  a  time.  However,  in  all  cases,  when  we  detect  and  track  any  target,  we  verify  its 
accuracy  using  GT  data  automatically. 

For  the  mathematical  technique  development,  we  explore  the  wavelets  in  the 
determination  of  the  exact  angle  of  rotation  between  the  images.  We  compare  the  performance  of 
the  Hilbert  transform  only  and  Hilbert  wavelet  transform  in  the  determination  of  the  exact  angle 
of  rotation  between  the  images.  Note  that  this  Hilbert-wavelet  analysis  is  performed  as  part  of 
our  other  ongoing  research  in  the  relevant  areas.  Further,  to  enhance  the  speed  of  operation  as 
well  as  data  reduction,  we  have  developed  a  zonal  mask  that  discards  up  to  50%  of  the 
insignificant  transformed  coefficients  while  retaining  most  of  the  information.  In  the 
determination  of  in-plane  rotation,  the  performance  of  the  Hilbert  wavelet  and  the  Hilbert 
transform  is  comparable.  However,  for  the  out-of-plane  rotation  case,  Hilbert  wavelet  offers 
improved  performance  with  a  less  number  of  false  alarms.  Detection  and  tracking  performance 
with  Hilbert  transform  and  Hilbert  wavelet  are  the  same  except  the  fact  that  Hilbert  wavelet 
offers  greater  correlation  value  and  thereby  improve  tracking.  Further,  we  have  developed  very 
flexible  GUI  that  allows  for  integration  for  not  our  algorithms,  by  also  additional  relevant 
algorithms  quite  easily.  This  GUI  may  be  instrumental  in  automated  comparison  and  decision 
fusion  of  algorithms  for  better  target  detection  and  tracking. 

We  take  advantage  of  our  novel  pre-processing,  detection  and  tracking  algorithms  in 
discarding  unwanted  frames  in  a  sequence  of  images.  This  improves  the  subsequent 
classification  performance  of  our  classifier.  We  perform  block  processing  on  the  remaining  pre- 
processed  and  noiseless  frames  and  extract  various  statistical  intensity  features  such  as  the  mean, 
variance,  minimum  and  maximum  intensity  values  of  the  image  from  every  block  and  use  them 
as  feature  vectors  for  classification.  We  propose  an  algorithm  to  extract  shape  features  of  the 
object  based  on  the  clean  edges  of  an  image.  We  have  proposed  an  edge-tracing  algorithm  to 
trace  the  coordinates  of  the  edge  points  of  the  image.  We  render  the  edges  invariant  by  shifting 
them  to  the  centroid  of  the  image.  We  sample  it  to  have  edges  of  the  order  of  powers  of  2.  We 
take  the  first  few  DCT  coefficients  to  represent  the  shape  and  make  them  invariant  to  scale. 
Finally,  we  use  the  reduced  number  of  shape-related  DCT  coefficients  as  the  input  for 
classification. 

We  use  self-organizing  maps  (SOM)  as  the  classification  algorithm  as  we  consider  the 
data  to  be  unlabeled.  The  usage  of  SOM  classification  offers  a  reasonable  number  of  clusters 
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(average  of  7  or  8)  for  400  frames  for  an  example  sequence  from  image  database.  In  addition, 
detection  and  classification  of  infrared  images  is  performed  using  the  Bayesian  conditional 
probability  algorithm  and  K-  nearest  neighbor  (NN)  classifier.  We  also  investigate  the  sensitivity 
of  the  algorithm  to  different  parametric  variations.  Various  examples  illustrating  the  limitation 
have  been  shown.  From  the  results,  it  is  clear  that  the  statistical  features  offer  better  classification 
results.  However,  we  must  emphasize  that  in  order  to  obtain  a  reasonably  accurate  classification 
of  data,  we  require  robust  reference  (example)  images  for  training  and  target  class  identification. 
The  current  dataset  provided  by  AMCOM  do  not  contain  such  ‘pristine’  reference  targets.  Thus, 
all  these  classification  results  in  this  report  are  intended  to  demonstrate  the  possible  directions  in 
the  event  if  such  reference  images  are  available. 
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Evaluating  Phase-Only  Filter  Algorithm  Performance  on  ARO  IR  Image  Sequences  Using 
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Executive  summary 

This  report  summarizes  the  work  performed  at  Wright  State  University.  The  Quality  Metric 
Scene  Evaluator  (QMSE),  a  front  end  GUI  and  the  search  engine  that  was  implemented  earlier 
during  this  project  was  used  to  process  the  data.  Trying  to  visually  determining  data  quality  is 
misleading1,  therefore,  the  ATR  performance  comparison  is  measured  with  the  quantitative 
knowledge  provided  by  the  ground  truth  data  provided.  Visual  inspection  on  the  other  hand  can 
provide  insight  while  debugging  and  improving  algorithms. 

The  search  engine  design  allows  for  easy  plug  in  of  multiple  search  methods.  Therefore,  scenes 
can  be  evaluated  based  upon  the  performance  of  different  matching  algorithms.  The  key  idea  of 
this  search  method  is  to  take  advantage  of  the  "divide  and  conquer"  concept.  Instead  of  searching 
for  a  pattern  in  a  large  image,  a  smart  approach  is  taken  to  divide  the  image  space  into 
overlapping  pattern  of  sub-images.  Search  is  then  based  on  upon  best  match  with  sub-image.  The 
QMSE  uses  the  Ground  Truth  Data  (GTD),  supplied  by  ARO,  to  compare  actual  target  location 
to  the  location  determined  by  the  tracker. 


Introduction 

This  report  is  the  final  report  from  Wright  State  University  on  the  ARO  project.  Progress  made 
during  the  last  trimester  of  2003  is  discussed.  The  direction  of  the  project  at  Wright  State 
University  shifted  by  the  year-end  2002,  from  one  exploring  and  designing  platform  to  evaluate 
different  search  algorithms,  to  the  one  of  actually  developing  framework  for  performing 
experiments.  ARO  provided  files  containing  Ground  Truth  data  information  to  aid  in  this  effort. 

Quality  Metric  Scene  Evaluator  (QMSE) 

During  second  quarter  a  Graphical  User  Interface  (GUI)  was  developed  to  ease  the  task  of 
evaluating  different  scenes,  using  different  algorithms.  This  is  the  front  end  of  the  search  engine 
designed  and  developed  at  WSU  earlier  to  evaluate  images.  Still  only  a  phase  only  filter  is 
available  to  track  objects  in  the  scenes,  but  with  two  different  algorithms  to  define  area  of 
interest.  However,  the  search  engine  design  allows  for  easy  plug  in  of  other  search  methods. 
Therefore,  scenes  can  be  evaluated  based  upon  the  performance  of  different  search  algorithms. 

The  key  idea  of  this  framework  is  to  take  advantage  of  the  "divide  and  concur"  concept.  Instead 
of  searching  for  a  pattern  in  a  large  image,  a  smart  approach  is  taken  to  divide  the  image  space 
into  overlapping  pattern  of  sub-images.  Search  is  then  based  upon  best  match  of  input  target  with 
sub-image.  Adaptive  algorithm  using  POF  was  designed  for  this  purpose  during  the  first  phase. 
The  algorithm  takes  target  sub-image  of  previous  scene  as  the  search  pattern  for  the  next  scene. 
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The  search  engine  framework  is  capable  of  dividing  image  space  into  sub-images  of  64x64, 
32x32,  16x16,  and  8x8  pixels.  This  feature  allows  experimenter  to  evaluate  susceptibility  of 
different  search  algorithm  to  the  speed  of  target,  background  noise,  frame  losses  and  other 
performance  issues  of  image  quality. 

Data  Analysis 

To  facilitate  automatic  processing  of  the  scenes,  an  extensive  analysis  of  the  ground  truth  data 
was  needed.  During  the  first  attempt  processing  the  data  (spring  and  summer  quarter)  only  the 
first  target  in  each  scene  was  tracked.  During  that  phase  it  came  apparent  that  the  complexity  of 
the  different  scenes  and  the  variety  of  data  format  in  the  ground  truth  files  opposed  a  problem 
during  processing.  The  vast  number  of  ground  truth  data  files  (approximately  20k  files)  and  the 
variety  of  situations  in  the  scenes  made  this  a  challenging  task.  In  order  to  understand  the 
underlying  requirements  for  the  search  engine  program,  a  Matlab  program  was  written  to 
generate  data  file  which  then  was  imported  into  spreadsheet  program  for  analysis.  From  the  fifty 
sequences  less  than  half  had  only  one  target  present.  One  sequence  had  five  targets  present  in  the 
scene  at  one  time  although  all  in  all  there  were  more  than  five  targets  in  the  scene  at  different 
times  Figure  1. 


ARO  Data 


Sequence 


Figure  1  Number  of  targets  at  one  time  in  a  sequence 

The  following  charts,  figures  2  through  8,  illustrate  the  wide  variety  of  scenes  encountered  in  the 
sequences.  Sequence  L1415  belongs  to  the  simplest  of  data.  Only  one  target  present  from  the 
beginning  to  the  end,  starting  in  frame  one. 
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L1415 


Frame 


Figure  2  Simple  sequence 

Figure  2,  represents  one  of  the  simplest  sequences.  Another  simple  sequence  follows  in  figure  3. 
Sequence  LI  520  is  still  a  sequence  of  only  one  target,  but  the  ground  truth  data  not  available  for 
the  first  44  files. 

LI  520 


Frame 


Figure  3  Little  more  complex  data 

Still  the  complexity  increases,  or  the  variety  of  processing  requirements.  In  figure  4  sequence 
L15NS  we  have  a  sequence  that  starts  out  with  a  one  target  and  then  reduces  to  only  one  target 
and  stays  there  for  the  reminder  of  the  sequence. 
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L15NS 


Frame 


Figure  4  Still  increasing  complexity 

Some  sequences  have  variable  number  of  target  present  over  the  course  of  the  scene  but  are  still 
relatively  easy  to  process  automatically,  see  figure  5. 


L1812 


Frame 


Figure  5  Still  relatively  simple  to  process 

But  the  complexity  in  automatic  processing  increases  as  the  variety  of  the  format  increases.  Few 
of  the  more  difficult  sequences  follow. 
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L1805 


Frame 


Figure  6  Number  of  targets  in  the  scene  changes  multiple  times 

Sequence  LI 702  is  one  of  the  more  challenging  sequences  counting  seven  different  targets 
although  there  are  maximum  three  visible  at  any  given  time. 


LI  702 


Frame 


Figure  7  Missing  initial  ground  truth  data  and  varying  number  of  targets 
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L19NS 


Frame 


Figure  8  Missing  ground  truth  data  (target)  in  the  middle  of  a  sequence 

A  complete  set  of  ground  truth  data  tables  and  graphs  is  provided  in  the  appendix.  Due  to  the 
variety  of  the  data  a  database  was  constructed  including  all  the  given  ground  truth  data.  Using 
this  database,  a  lookup  table  is  created  for  bookkeeping  of  the  targets  in  each  scene.  Moreover, 
the  multiple  mat-fdes  for  each  sequence  were  combined  into  one  file  per  sequence. 


Scene  Evaluation 

Using  the  Phase  Only  Filter  (POF)  two  experiments  were  run  on  all  the  image  sequences.  One 
experiment  uses  Gaussian  weighting  function  and  the  other  experiment  uses  a  distance  function 
to  define  the  region  of  interest.  The  functionality  and  operator  instructions  for  the  QMSE  can  be 
found  in  the  June  2003  report  as  well  as  in  the  appendix  of  this  report.  Trying  to  visually 
determining  data  quality  is  misleading1,  therefore,  the  ATR  performance  is  measured  only  with 
the  quantitative  knowledge  gained  from  the  ground  truth  data  provided.  Visual  inspection  on  the 
other  hand  can  provide  insight  while  debugging  and  improving  the  algorithms. 

The  QMSE  reads  data  from  the  Ground  Truth  Data-base  and  uses  this  data  to  compare  actual 
target  location  to  the  location  determined  by  the  tracker.  The  QMSE  pumps  out  information  to 
the  user  interface  and  to  a  text  file  regarding  the  tracking  progress.  If  the  target  moves  out  of  the 
tracking  gate  an  error  message  is  displayed  and  this  data  is  used  to  calculate  the  success  rate.  The 
success  rate  is  determined  by: 


SuccessRate  =  100- 


miss 

number  _of  _  frames 
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It  should  be  noted  that  the  variable  number  of  Jrames  is  not  equal  to  the  number  of  frames  in  the 
sequence.  The  variable  refers  to  the  number  of  frames  were  the  target  is  present  according  to  the 
ground  truth  data  files. 

The  first  time  a  target  appears  the  ground  truth  data  is  used  to  obtain  the  reference  target.  This  is 
the  only  time  ground  truth  data  is  used  to  provide  tracking  data  for  the  algorithm.  Following  this 
initial  extraction  of  the  target  the  ground  truth  data  provides  quantitative  data  to  measure  the 
performance  of  the  algorithm. 

It  is  relatively  easy  to  add  different  search  algorithms,  weighting  functions  and  preprocessing 
functionality  to  the  QMSE.  The  user  interface  provides  the  entry  hook  and  the  function  (search 
algorithm)  can  be  added  as  a  new  separate  “m”  file  to  the  project.  A  switch  statement  would  be 
needed  in  the  main  loop  to  accommodate  multiple  search  algorithms.  Pre  processing  would  be 
added  in  the  outer  loop  right  after  the  “mat”  file  containing  the  sequence  is  opened. 

Some  experimentation  with  the  weighting  function  might  provide  improved  performance. 
However,  tweaking  of  the  weighting  function  requires  the  entire  data  set  to  be  evaluated  each 
time  to  avoid  tailoring  the  function  to  certain  scenes  or  targets. 

When  working  with  multiple  targets  the  QMSE  displays  a  number  next  to  the  gate  (sub-image). 
This  number  refers  to  the  target  that  is  being  tracked.  The  name  of  the  target  is  presented  in  the 
Ground  truth  data  area.  For  example:  if  there  are  three  active  targets,  tank,  truck,  and  Bradley 
appearing  in  this  order  in  the  ground  truth  data  file.  While  tracking  the  tank  a  number  “1”  will  be 
displayed  next  to  the  gate.  While  tracking  the  truck  a  “2”  will  be  displayed  and  so  on.  Rather 
than  running  the  same  sequence  multiple  times  when  tracking  multiple  targets,  the  tracker 
multiplexes  the  targets.  This  method  makes  it  easy  to  analyze  and  understand  why  the  algorithm 
fails.  In  some  cases  where  the  targets  are  initially  very  close  to  each  other  the  algorithm  can  fail 
to  track  the  correct  target  when  the  targets  separate. 
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Figure  9  QMSE  Gate  Labeling 

the  case  of  two  targets  close  together,  we  can  have  four  general  situations: 

1 .  The  algorithm  correctly  tracks  both  targets 

2.  The  algorithm  tracks  the  same  target  twice 

3.  The  tracker  assigned  initially  to  target  one  actually  tracks  target  number  two  and  the 
tracker  assigned  to  target  two  tracks  target  number  one. 

4.  Not  tracking  at  all,  lost. 
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Figure  10  QMSE  Gate  label 


Of  course  this  problem  compounds  when  the  number  of  targets  increases. 
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Processing  the  scene 
Sub-Imaging 

An  image  can  be  represented  by  the  following  set: 


Im  =  {/>!  i ,  p{  2  , . p, 28>i28  } 


(1) 


Similarly  the  target  can  be  represented  by  the  set: 

T  ~  {h,l’h,2’’"hnax,max} 


(2) 


We  know  that  target  T  is  embedded  in  the  scene  Im  since  Jis  a  proper  subset  of  Im: 

T  e  Im  (3) 

This  leads  to  one  task  only;  the  task  of  finding  the  subset  of  Im  (sub-image)  containing  the  target 
T.  Such  that: 

TeS  Im  c  Im  (4) 

To  focus  on  features  rather  than  the  energy  of  the  image  we  must  establish  a  reference  value. 
Sub-imaging  and  comparing  can  select  the  best  match  as  being: 

T  ~  S  Im  c:  Im  (5) 

Therefore,  the  search  algorithm  is  precisely  as  follows: 

\/SIm®T,3SIm®T\SIm®T  *T®T  (6) 


This  search  algorithm  truly  eliminates  the  effect  of  high-energy  pixels,  and  the  POF  searches  for 
the  best  matching  features.  This  algorithm  can  be  stated  as:  For  all  sub-images  convolved  with 
target  image,  there  must  exist  sub-image  convolved  with  target  image  such  that  sub-image 
convolved  with  target  image  is  approximately  same  as  target  image  convolved  with  target  image. 
So  the  task  on  hand  is  to  find  the  cross  correlation  of  sub-image  with  target  image  that  is  closest 
in  magnitude  to  the  autocorrelation  of  the  target  image. 

A  simple  checkerboard  sub-imaging  is  not  sufficient  to  produce  reliable  results.  A  more 
sophisticated  approach  must  be  taken  in  order  to  preserve  information  between  scenes.  Figure  1 1 
shows  one  such  approach.  Sub-images  are  created  in  overlapping  fashion. 
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Figure  1 1  Sub-image  grid 


One  of  the  most  important  parameters,  to  be  considered  when  selecting  granularity  of  the  sub¬ 
imaging,  is  the  size  of  the  target.  The  smaller  the  target  the  smaller  the  sub-image  must  be. 
However,  to  ensure  maximum  detection  capability  the  target  size  needs  to  be  %  of  the  sub-image 
size.  If  the  sub-image  is  smaller  than  the  target  a  maximum  pattern  matching  will  not  be 
achieved  due  to  clipping.  If  the  sub-image  on  the  other  hand  is  bigger  than  the  target  some  of  the 
background  will  be  included,  and  the  edge  detection  weight  of  the  POF  might  be  overcome  by 
the  weight  of  the  energy  summed  up  by  the  integration2. 

Algorithm 

Figure  1 1  describes  the  adaptive  POF  ATR  system.  A  target  is  extracted  at  its  first  appearance  in 
the  input  image.  The  target  along  with  the  image  frame  is  passed  on  to  the  POF  ATR  system. 
The  POF  ATR  System  consists  of  four  main  modules:  The  module  extracting  sub-images  from 
the  input  image,  auto-correlator  module  to  establish  the  reference  value  for  match,  correlator 
module  which  correlates  each  sub-image  to  the  input  target,  and  finally  a  selector  module  to 
select  the  target  sub-image.  In  this  research  we  have  provided  results  from  selector  module  using 
two  different  weighting  functions.  One  is  based  on  the  Gaussian  weight,  and  the  other  method  is 
based  on  actual  distance  from  the  center  of  target  area.  The  selector  feeds  back  the  new  target  to 
the  auto-correlator  module  to  update  the  input  target,  and  to  the  output  for  displaying.  Next  it 
updates  the  target  and  feeds  it  back  to  the  auto-correlator  to  be  used  as  an  input  filter  for  the  next 
image  frame.  Finally  it  passes  information  on  to  the  output  module  to  display  the  tracking 
results. 
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Figure  12  POF  Tracking  System 

The  ATR  algorithm  performance  is  clearly  directly  related  to  the  target  signature  and  the  clutter 
of  the  scene3.  Providing  the  two  different  weighting  methods  for  the  selector  module  clearly 
shows  this. 


Results 

All  102  targets  were  tracked  automatically  and  evaluated  based  on  the  ground  truth  data  provided 
by  ARO  for  two  different  weighting  functions.  The  results  are  provided  in  tables  1  and  2.  Figure 
13  compares  the  two  weighting  functions. 


Gaussian  vs.  Distance 


- Gaussian 

Distance 


Target 


Figure  13  Performance  comparison  of  Gaussian  vs.  Distance  weight  function 

Following  table  shows  the  results  from  tracking  the  targets  using  weighting  function  based  on  the 
Gaussian  distribution.  In  the  tables  1  and  2,  the  values  of  min,  mean,  max,  and  std  represent  the 
average  values  over  the  sequence.  The  miss  value  reflects  the  total  number  of  frames  missed 
while  tracking,  irrespective  of  the  location  of  the  frame  in  the  sequence. 
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Table  1  Results  with  Gaussian  weighting  function. 


Sequence# 

min 

mean 

max 

std 

target 

misses 

#  of  scenes 

rate 

./Lwir/L14  _15/L14_15 

91 

133.92 

244 

3.34 

Mantruck 

0 

280 

100.00 

./Lwir/L15  20/Ll 5_20 

85 

132.25 

231 

2.95 

tankl 

95 

170 

44.12 

./Lwir/L15  NS/L15  NS 

84 

142.20 

225 

2.37 

tankl 

302 

319 

5.33 

./Lwir/L15  NS/L15  NS 

84 

142.20 

225 

2.37 

truck 

2 

21 

90.48 

./Lwir/L16  04/Ll 6  04 

83 

154.87 

249 

2.85 

tankl 

396 

399 

0.75 

./Lwir/L  1 6  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

185 

210 

11.90 

./Lwir/L  1 6  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

0 

188 

100.00 

./Lwir/L  1 6  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

0 

10 

100.00 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

M60 

42 

289 

85.47 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

apcl 

0 

81 

100.00 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

mantrk? 

44 

100 

56.00 

./Lwir/L  16  18/L16_18 

37 

124.18 

251 

4.57 

apcl 

264 

290 

8.97 

./Lwir/L  16  18/Ll  6  18 

37 

124.18 

251 

4.57 

M60 

0 

101 

100.00 

./Lwir/L  16  18/Ll  6  18 

37 

124.18 

251 

4.57 

truck 

0 

6 

100.00 

./Lwir/L  17  01/L17  01 

50 

127.49 

220 

2.26 

Bradley? 

0 

370 

100.00 

./Lwir/L  1 7  01/L17  01 

50 

127.49 

220 

2.26 

pickup(trk)? 

12 

43 

72.09 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Mantruck? 

32 

100 

68.00 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

pickup(trk)? 

0 

32 

100.00 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Bradley? 

325 

340 

4.41 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Mantruck? 

158 

418 

62.20 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

tank 

16 

48 

66.67 

./Lwir/L  17  20/L17  20 

44 

126.33 

240 

7.87 

target 

34 

35 

2.86 

./Lwir/L  17  20/Ll 7  20 

44 

126.33 

240 

7.87 

M60 

591 

734 

19.48 

./Lwir/L  18  03/Ll 8  03 

38 

117.98 

226 

4.87 

Bradley 

389 

446 

12.78 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

tankl 

605 

747 

19.01 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

apcl 

44 

135 

67.41 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

M60 

104 

159 

34.59 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

tank 

8 

9 

11.11 

./Lwir/L  18  07/Ll 8  07 

66 

156.13 

241 

6.20 

Bradley 

241 

259 

6.95 

./Lwir/L  18  12/Ll  8  12 

51 

142.46 

231 

5.64 

Bradley 

94 

299 

68.56 

./Lwir/L  18  12/L18_12 

51 

142.46 

231 

5.64 

tankl 

167 

198 

15.66 

./Lwir/L  18  12/L18  12 

51 

142.46 

231 

5.64 

M60 

0 

92 

100.00 

./Lwir/L  1 8  13/L18  13 

55 

143.89 

235 

5.52 

tankl 

292 

325 

10.15 

./Lwir/L  18  13/Ll  8  13 

55 

143.89 

235 

5.52 

M60 

0 

228 

100.00 

./Lwir/L  18  13/Ll  8  13 

55 

143.89 

235 

5.52 

apcl 

0 

168 

100.00 

./Lwir/L  1 8  15/Seql/L18  15 

54 

142.57 

250 

5.59 

M60 

0 

242 

100.00 

./Lwir/L  1 8  15/Seql/L18  15 

54 

142.57 

250 

5.59 

truck? 

0 

12 

100.00 

./Lwir/L  18  15/Seql/L18  15 

54 

142.57 

250 

5.59 

tankl 

262 

288 

9.03 

./Lwir/L  18  15/Seql/L18  15 

54 

142.57 

250 

5.59 

bradley 

22 

192 

88.54 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

M60 

206 

237 

13.08 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

tankl 

265 

306 

13.40 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

bradley 

88 

186 

52.69 

./Lwir/L  18  1 6/L 18  16 

48 

142.48 

243 

5.93 

tankl 

300 

327 

8.26 

./Lwir/L  18  16/Ll  8  16 

48 

142.48 

243 

5.93 

M60 

0 

289 

100.00 

./Lwir/L  1 8  17/Seql/L18  17 

51 

150.07 

252 

5.88 

tankl 

172 

239 

28.03 

./Lwir/L  1 8  17/Seql/L18  17 

51 

150.07 

252 

5.88 

M60 

0 

194 

100.00 

66 


./Lwir/L18  17/Seq2/L18  17 

47 

148.04 

252 

6.04 

tankl 

200 

265 

24.53 

./Lwir/L18  17/Seq2/L18  17 

47 

148.04 

252 

6.04 

M60 

0 

190 

100.00 

./Lwir/L18  18/L18_18 

52 

146.41 

252 

5.70 

mantruck 

43 

45 

4.44 

./Lwir/L18  18/L18_18 

52 

146.41 

252 

5.70 

target 

1 

4 

75.00 

./Lwir/L18  18/L18_18 

52 

146.41 

252 

5.70 

apcl 

52 

92 

43.48 

./Lwir/L  181 8/L 1818 

52 

146.41 

252 

5.70 

M60? 

114 

122 

6.56 

./Lwir/L18  18/Ll  8  18 

52 

146.41 

252 

5.70 

tankl 

203 

213 

4.69 

./Lwir/L  18  18/L18_18 

52 

146.41 

252 

5.70 

testvan 

0 

3 

100.00 

./Lwir/L  1 9  01/L19  01 

54 

143.46 

252 

5.90 

tankl 

210 

239 

12.13 

./Lwir/L  19  02/Ll 9  02 

53 

143.93 

252 

5.88 

tankl 

201 

269 

25.28 

./Lwir/L  19  04/Ll 9  04 

43 

140.32 

252 

6.74 

tankl 

186 

269 

30.86 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

tankl 

234 

264 

11.36 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

apcl 

202 

264 

23.48 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

Mantruck 

0 

205 

100.00 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

Van 

0 

22 

100.00 

./Lwir/L  19  07/Ll 9  07 

56 

151.43 

253 

5.53 

Mantruck 

57 

75 

24.00 

./Lwir/L  19  10/Ll  9  10 

54 

148.14 

248 

5.40 

tankl 

0 

113 

100.00 

./Lwir/L  19  10/Ll  9  10 

54 

148.14 

248 

5.40 

apcl 

68 

73 

6.85 

./Lwir/L  19  1 1/L191 1 

51 

142.89 

253 

5.41 

tankl 

5 

164 

96.95 

./Lwir/L  19  1 1/L191 1 

51 

142.89 

253 

5.41 

apcl 

0 

164 

100.00 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

tankl 

121 

264 

54.17 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

apcl 

238 

264 

9.85 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

M60 

0 

66 

100.00 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

Mantruck 

1 

12 

91.67 

./Lwir/L  19  15/L19  15 

39 

140.70 

255 

5.10 

tankl 

306 

321 

4.67 

./Lwir/L  19  15/L19_15 

39 

140.70 

255 

5.10 

apcl 

3 

190 

98.42 

./Lwir/L  19  15/L19  15 

39 

140.70 

255 

5.10 

M60 

319 

326 

2.15 

./Lwir/L  19  15/Ll  9  15 

39 

140.70 

255 

5.10 

Van 

19 

210 

90.95 

./Lwir/L  19  15/Ll  9  15 

39 

140.70 

255 

5.10 

apcl 

9 

12 

25.00 

./Lwir/L  19  18/L19_18 

41 

144.89 

252 

4.74 

tankl 

141 

233 

39.48 

./Lwir/L  19  18/L19_18 

41 

144.89 

252 

4.74 

M60? 

0 

12 

100.00 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

M60? 

5 

58 

91.38 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

mantruck 

77 

114 

32.46 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

tankl 

87 

128 

32.03 

./Lwir/L20  04/L20  04 

44 

139.58 

230 

3.36 

apcl 

0 

367 

100.00 

./Lwir/L20  08/L20  08 

44 

143.50 

236 

3.93 

apcl 

164 

347 

52.74 

./Lwir/L20_l 7/L201 7 

42 

141.03 

233 

3.77 

tankl 

123 

191 

35.60 

./Lwir/L20  18/L20  18 

43 

138.61 

224 

3.30 

tankl 

0 

447 

100.00 

./Lwir/L20  20/L20  20 

50 

140.40 

216 

3.14 

tankl 

192 

419 

54.18 

.  /L  wir/L2 0  2 0/L2 0  2 0 

50 

140.40 

216 

3.14 

target 

111 

172 

35.47 

./Lwir/L21  04/L21  04 

55 

144.01 

183 

1.37 

bradley 

319 

321 

0.62 

./Lwir/L21  04/L21  04 

55 

144.01 

183 

1.37 

tankl 

386 

690 

44.06 

./Lwir/L2 1  15/L21  15 

54 

140.29 

194 

2.08 

Bradley 

731 

737 

0.81 

./Lwir/L2 1  17/L21  17 

54 

136.28 

196 

2.04 

apcl 

0 

359 

100.00 

./Lwir/L22  06/L22  06 

51 

139.39 

211 

2.52 

tankl 

0 

347 

100.00 

./Lwir/L22  08/L22  08 

45 

130.85 

212 

2.98 

apcl 

0 

379 

100.00 

./Lwir/L22  14/L22  14 

47 

134.95 

209 

2.71 

M60 

270 

273 

1.10 

./Lwir/L22  14/L2214 

47 

134.95 

209 

2.71 

target 

72 

74 

2.70 

67 


./Lwir/L23  12/L23  12 

49 

134.09 

216 

2.46 

apcl 

0 

367 

100.00 

./Mwir/M14  06/M14  06 

35 

141.62 

216 

2.44 

Bradley 

0 

378 

100.00 

./Mwir/M14  07/M14  07 

34 

143.36 

218 

2.39 

Bradley 

0 

399 

100.00 

./Mwir/M14  10/M14  10 

25 

125.03 

228 

3.71 

tankl 

0 

497 

100.00 

./Mwir/M14  10/M14  10 

25 

125.03 

228 

3.71 

Mantruck 

14 

77 

81.82 

./Mwir/M  1 41 3/M  1413 

25 

126.70 

228 

4.04 

Mantruck 

0 

379 

100.00 

./Mwir/M  1 4  1 5/M  1 4  1 5 

25 

138.22 

228 

3.00 

Mantruck 

513 

523 

1.91 

./Mwir/M  14  15/M14 15 

25 

138.22 

228 

3.00 

target 

11 

19 

42.11 

The  following  bar  graph  summarizes  the  results  using  the  Gaussian  weighting  function. 
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Figure  14  Results  using  Gaussian  weighting  function 


The  results  using  the  Distance  weighting  function  follows  in  table  2. 


Table  2  Results  using  Distance  weighting  function. 


Sequence# 

min 

mean 

max 

std 

target 

misses 

#  of  scenes 

rate 

./Lwir/L14  15/L14  15 

91 

133.92 

244 

3.34 

Mantruck 

0 

280 

100.00 

./Lwir/L15  20/Ll 5  20 

85 

132.25 

231 

2.95 

tankl 

97 

170 

42.94 

./Lwir/L15  NS/L15  NS 

84 

142.20 

225 

2.37 

tankl 

299 

319 

6.27 

./Lwir/L  1 5  NS/L15  NS 

84 

142.20 

225 

2.37 

truck 

1 

21 

95.24 

./Lwir/L16  04/Ll 6  04 

83 

154.87 

249 

2.85 

tankl 

396 

399 

0.75 

./Lwir/L  1 6  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

192 

210 

8.57 

./Lwir/L  16  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

0 

188 

100.00 

./Lwir/L  16  07/Ll 6  07 

81 

140.98 

251 

2.86 

tankl 

0 

10 

100.00 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

M60 

46 

289 

84.08 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

apcl 

0 

81 

100.00 

./Lwir/L  16  08/Ll 6  08 

47 

111.72 

255 

4.24 

mantrk? 

45 

100 

55.00 

./Lwir/L  16  18/L16 18 

37 

124.18 

251 

4.57 

apcl 

265 

290 

8.62 

./Lwir/L  16  18/Ll  6  18 

37 

124.18 

251 

4.57 

M60 

0 

101 

100.00 

./Lwir/L  16  18/Ll  6  18 

37 

124.18 

251 

4.57 

truck 

0 

6 

100.00 

68 


./Lwir/L17  01/L17  01 

50 

127.49 

220 

2.26 

Bradley? 

0 

370 

100.00 

./Lwir/L17  01/L17  01 

50 

127.49 

220 

2.26 

pickup(trk)? 

12 

43 

72.09 

./Lwir/L17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Mantruck? 

32 

100 

68.00 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

pickup(trk)? 

0 

32 

100.00 

./Lwir/L17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Bradley? 

327 

340 

3.82 

./Lwir/L  17  02/Ll 7  02 

55 

147.29 

229 

2.09 

Mantruck? 

416 

418 

0.48 

./Lwir/L17  02/Ll 7  02 

55 

147.29 

229 

2.09 

tank 

34 

48 

29.17 

./Lwir/L  17  20/Ll 7  20 

44 

126.33 

240 

7.87 

target 

34 

35 

2.86 

./Lwir/L  17  20/Ll 7  20 

44 

126.33 

240 

7.87 

M60 

719 

734 

2.04 

./Lwir/L  18  03/Ll 8  03 

38 

117.98 

226 

4.87 

Bradley 

395 

446 

11.43 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

tankl 

680 

747 

8.97 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

apcl 

44 

135 

67.41 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

M60 

128 

159 

19.50 

./Lwir/L  18  05/Ll 8  05 

67 

163.61 

248 

6.32 

tank 

8 

9 

11.11 

./Lwir/L  18  07/Ll 8  07 

66 

156.13 

241 

6.20 

Bradley 

241 

259 

6.95 

./Lwir/L  18  12/Ll  8  12 

51 

142.46 

231 

5.64 

Bradley 

2 

299 

99.33 

./Lwir/L  18  12/Ll  8  12 

51 

142.46 

231 

5.64 

tankl 

169 

198 

14.65 

./Lwir/L  18  12/L18  12 

51 

142.46 

231 

5.64 

M60 

0 

92 

100.00 

./Lwir/L  18  13/Ll  8  13 

55 

143.89 

235 

5.52 

tankl 

292 

325 

10.15 

./Lwir/L  18  13/Ll  8  13 

55 

143.89 

235 

5.52 

M60 

0 

228 

100.00 

./Lwir/L  18  13/Ll  8  13 

55 

143.89 

235 

5.52 

apcl 

0 

168 

100.00 

./Lwir/L  18  15/Seql/L18  15 

54 

142.57 

250 

5.59 

M60 

187 

242 

22.73 

./Lwir/L  1 8  15/Seql/L18  15 

54 

142.57 

250 

5.59 

truck? 

0 

12 

100.00 

./Lwir/L  1 8  15/Seql/L18  15 

54 

142.57 

250 

5.59 

tankl 

261 

288 

9.38 

./Lwir/L  18  15/Seql/L18  15 

54 

142.57 

250 

5.59 

bradley 

79 

192 

58.85 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

M60 

199 

237 

16.03 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

tankl 

263 

306 

14.05 

./Lwir/L  18  15/Seq2/L18  15 

49 

139.27 

227 

5.43 

bradley 

91 

186 

51.08 

./Lwir/L  18  16/Ll  8  16 

48 

142.48 

243 

5.93 

tankl 

314 

327 

3.98 

./Lwir/L  18  16/Ll  8  16 

48 

142.48 

243 

5.93 

M60 

0 

289 

100.00 

./Lwir/L  18  17/Seql/L18  17 

51 

150.07 

252 

5.88 

tankl 

168 

239 

29.71 

./Lwir/L  18  17/Seql/L18  17 

51 

150.07 

252 

5.88 

M60 

0 

194 

100.00 

./Lwir/L  18  17/Seq2/L18  17 

47 

148.04 

252 

6.04 

tankl 

204 

265 

23.02 

./Lwir/L  18  17/Seq2/L18  17 

47 

148.04 

252 

6.04 

M60 

0 

190 

100.00 

./Lwir/L  18  18/L18_18 

52 

146.41 

252 

5.70 

mantruck 

41 

45 

8.89 

./Lwir/L  18  18/L18_18 

52 

146.41 

252 

5.70 

target 

2 

4 

50.00 

./Lwir/L  1 8  1 8/L 1818 

52 

146.41 

252 

5.70 

apcl 

71 

92 

22.83 

./Lwir/L  18  18/Ll  8  18 

52 

146.41 

252 

5.70 

M60? 

114 

122 

6.56 

./Lwir/L  18  18/Ll  8  18 

52 

146.41 

252 

5.70 

tankl 

199 

213 

6.57 

./Lwir/L  181 8/L  1818 

52 

146.41 

252 

5.70 

testvan 

0 

3 

100.00 

./Lwir/L  19  01/L19  01 

54 

143.46 

252 

5.90 

tankl 

195 

239 

18.41 

./Lwir/L  19  02/Ll 9  02 

53 

143.93 

252 

5.88 

tankl 

230 

269 

14.50 

./Lwir/L  19  04/Ll 9  04 

43 

140.32 

252 

6.74 

tankl 

182 

269 

32.34 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

tankl 

233 

264 

11.74 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

apcl 

202 

264 

23.48 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

Mantruck 

0 

205 

100.00 

./Lwir/L  19  06/Ll 9  06 

55 

149.00 

252 

5.35 

Van 

0 

22 

100.00 

./Lwir/L  19  07/Ll 9  07 

56 

151.43 

253 

5.53 

Mantruck 

50 

75 

33.33 

69 


./Lwir/L19  10/Ll  9  10 

54 

148.14 

248 

5.40 

tankl 

0 

113 

100.00 

./Lwir/L19  10/L19_10 

54 

148.14 

248 

5.40 

apcl 

0 

73 

100.00 

./Lwir/L  1 91 1/L19  11 

51 

142.89 

253 

5.41 

tankl 

0 

164 

100.00 

./Lwir/L19  1 1/L191 1 

51 

142.89 

253 

5.41 

apcl 

0 

164 

100.00 

./Lwir/L  1 9  13/Ll  9  13 

54 

152.68 

253 

5.20 

tankl 

118 

264 

55.30 

./Lwir/L  1 9  13/L19  13 

54 

152.68 

253 

5.20 

apcl 

261 

264 

1.14 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

M60 

1 

66 

98.48 

./Lwir/L  19  13/Ll  9  13 

54 

152.68 

253 

5.20 

Mantruck 

1 

12 

91.67 

./Lwir/L  19  15/Ll  9  15 

39 

140.70 

255 

5.10 

tankl 

303 

321 

5.61 

./Lwir/L  19  15/Ll  9  15 

39 

140.70 

255 

5.10 

apcl 

3 

190 

98.42 

./Lwir/L  19  15/L19  15 

39 

140.70 

255 

5.10 

M60 

319 

326 

2.15 

./Lwir/L  19  15/L19_15 

39 

140.70 

255 

5.10 

Van 

23 

210 

89.05 

./Lwir/L  19  15/Ll  91 5 

39 

140.70 

255 

5.10 

apcl 

9 

12 

25.00 

./Lwir/L  19  18/Ll  9  18 

41 

144.89 

252 

4.74 

tankl 

26 

233 

88.84 

./Lwir/L  19  18/Ll  9  18 

41 

144.89 

252 

4.74 

M60? 

0 

12 

100.00 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

M60? 

3 

58 

94.83 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

mantruck 

100 

114 

12.28 

./Lwir/L  19  NS/L19  NS 

38 

140.73 

252 

4.78 

tankl 

87 

128 

32.03 

./Lwir/L20  04/L20  04 

44 

139.58 

230 

3.36 

apcl 

0 

367 

100.00 

./Lwir/L20  08/L20  08 

44 

143.50 

236 

3.93 

apcl 

150 

347 

56.77 

./Lwir/L20  17/L20_17 

42 

141.03 

233 

3.77 

tankl 

118 

191 

38.22 

./Lwir/L20  18/L20  18 

43 

138.61 

224 

3.30 

tankl 

0 

447 

100.00 

./Lwir/L20  20/L20_20 

50 

140.40 

216 

3.14 

tankl 

225 

419 

46.30 

./Lwir/L20  20/L20_20 

50 

140.40 

216 

3.14 

target 

138 

172 

19.77 

./Lwir/L21  04/L21  04 

55 

144.01 

183 

1.37 

bradley 

319 

321 

0.62 

./Lwir/L21  04/L21  04 

55 

144.01 

183 

1.37 

tankl 

385 

690 

44.20 

./Lwir/L21  15/L21_15 

54 

140.29 

194 

2.08 

Bradley 

694 

737 

5.83 

./Lwir/L2 1  17/L21  17 

54 

136.28 

196 

2.04 

apcl 

0 

359 

100.00 

./Lwir/L22  06/L22  06 

51 

139.39 

211 

2.52 

tankl 

0 

347 

100.00 

./Lwir/L22  08/L22  08 

45 

130.85 

212 

2.98 

apcl 

0 

379 

100.00 

./Lwir/L22  14/L22  14 

47 

134.95 

209 

2.71 

M60 

270 

273 

1.10 

./Lwir/L22  14/L22  14 

47 

134.95 

209 

2.71 

target 

72 

74 

2.70 

./Lwir/L23  12/L23  12 

49 

134.09 

216 

2.46 

apcl 

0 

367 

100.00 

./Mwir/M14  06/M  14  06 

35 

141.62 

216 

2.44 

Bradley 

0 

378 

100.00 

./Mwir/M14  07/M14  07 

34 

143.36 

218 

2.39 

Bradley 

0 

399 

100.00 

./Mwir/M14  10/M14_10 

25 

125.03 

228 

3.71 

tankl 

1 

497 

99.80 

./Mwir/M  1 4_  1 0/M  1410 

25 

125.03 

228 

3.71 

Mantruck 

14 

77 

81.82 

./Mwir/M14  13/M14_13 

25 

126.70 

228 

4.04 

Mantruck 

0 

379 

100.00 

./Mwir/M  14  15/M14_15 

25 

138.22 

228 

3.00 

Mantruck 

513 

523 

1.91 

./Mwir/M  14  15/M14_15 

25 

138.22 

228 

3.00 

target 

11 

19 

42.11 

The  following  bar  graph  summarizes  the  results  using  the  Distance  weighting  function. 
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Distance  weighting  function 


Figure  15  Results  using  Distance  weighting  function 


Conclusions 

In  this  report  we  have  discussed  the  status  of  the  ARO  project  at  WSU.  A  multi -target  tracking 
version  of  the  “Quality  Metric  Scene  Evaluator”  (QMSE)  has  been  developed  using  the  Matlab 
programming  tools.  Using  the  QMSE  we  have  evaluated  the  entire  data  set  supplied  by  the  Army 
Research  Office  twice  using  the  Phase  Only  Filter  in  combination  with  two  different  weighting 
functions,  Gaussian  function  and  distance  function.  The  QMSE  uses  sub-imaging  technique.  The 
sub-imaging  technique  shows  very  promising  results  when  used  in  conjunction  with  the  POF  and 
a  simple  hot-spot  filter.  Using  overlapping  sub-images  of  relevant  size  transforms  the  task  of 
tracking  a  target  in  a  scene  to  the  task  of  finding  the  best  matching  sub-image,  which  in  turn  has 
a  fixed  place  in  the  scene.  The  critical  parameter  here  is  to  select  the  sub-image  size  such  as  to 
maximize  the  target  occupation  in  the  sub-image.  Future  work  in  this  area  should  include 
applying  this  technique  to  other  ATR  methods.  Exploring  methods  of  expanding  and  shrinking 
the  grid  size  automatically  based  on  target  size  also  needs  to  be  explored. 
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Gaussian  weighting  performs  better  than  distance  weighting 
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Figure  16  Comparison  of  Gaussian  and  Distance  weighting  function 


The  QMSE  reads  data  from  the  Ground  Truth  Data  (GTD)  files  and  uses  this  data  to  compare 
actual  target  location  to  the  location  determined  by  the  tracker. 
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Appendix  A 

Quality  Metric  Scene  Evaluator 

The  following  figure  shows  the  QMSE  GUI.  Various  information  about  the  images  are 
displayed.  The  following  list  describes  each  item  of  the  GUI. 
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10 


2 


3. 


11 


4. 


12 


13 


5. 


6. 


14, 


7. 


15, 


16. 


8. 


17 


18 


9. 


1 .  Button  bar 

The  button  bar  has  five  buttons.  Only  two  have  been  implemented  so  far.  Those  are  the 
Start  button  and  the  Stop  button.  The  other  three  buttons  are  Pause,  and  buttons  for 
Forward  and  Backwards  stepping  one  frame  at  a  time. 

2.  Path 

The  field  below  the  button  bar  displays  the  relative  path  of  the  file  currently  being 
processed. 
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3.  Tracking  algorithm 

Pull  down  list  to  select  different  search  algorithms  to  evaluate. 

4.  Frame  Delay 

The  frame  delay  is  to  slow  down  or  speed  up  the  image  processing.  This  can  be  handy 
when  transitions  between  frames  need  to  be  examined  without  pausing  the  program. 

5.  Threshold 

The  threshold  slider  is  not  implemented  for  the  POF.  However,  it  is  provide  for 
algorithms  that  will  need  to  have  some  variable  parameter  such  as  threshold/ 

6.  Start  File 

Next  we  have  the  start  file  pull  down  list.  Any  of  the  files  provided  can  be  selected  to  be 
the  first  file  processed. 

NOTE: 

In  later  version  a  capability  to  process  the  files  in  backwards  order  will  be  provided. 

7.  Stop  File. 

The  stop  file  pull  down  list  serves  is  used  to  select  the  last  file  to  process. 

8.  Nearest  neighbours 

The  stem  diagram  shows  the  relationship  between  the  absolute  differences  of  the 
correlation  between  the  sub-images  in  the  region  of  interest. 

9.  Ground  Truth  data  display 

This  image  shows  the  sub-image  that  is  according  to  the  Ground  Truth  Data  files 
supposed  to  contain  the  target. 

10.  Ground  Truth  Data 

In  this  area  alphanumerical  information  from  the  Ground  Truth  files  are  displayed. 

1 1 .  Preprocessing. 

This  pull  down  list  provides  hook  to  plug  in  various  preprocessing  methods.  At  this  time 
no  preprocessing  is  provided. 

12.  Weighting  function 

The  current  POF  algorithm  can  use  either  a  Gaussian  function  to  determine  region  of 
interest  for  the  search  algorithm  or  a  plain  distance  function. 

13.  Sub-image  size. 

This  pull  down  list  allows  the  user  to  select  which  of  8x8,  16x16,  32x32,  64x64  sub¬ 
image  size  to  use  for  processing. 

14.  Tracking  Data. 

This  area  displays  alphanumerical  information  from  the  tracking  process. 

15.  Scene  Display. 

In  this  scene  display  the  new  target  area  is  displayed  in  the  current  frame. 

16.  Scene  Display. 

Here  the  new  target  is  displayed  in  the  new  frame. 

17.  Target  Display. 

In  this  target  display  the  new  target  is  displayed  in  the  current  frame. 

18.  Target  Display. 

Here  the  new  target  is  displayed  in  the  new  frame. 
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